field-service-agent/README.md

Field Service AI Agent - NPM package for integrating AI-powered field service management into mobile apps

# Field Service Agent Package

An NPM package for integrating AI-powered field service management into mobile applications. This package provides a natural language interface for managing clients, jobs, invoices, estimates, and communications in field service businesses.

## Features

- **Natural Language Processing**: Process voice or text commands to perform field service operations
- **Client Management**: Create, update, and retrieve client information
- **Job Management**: Create and manage work orders with scheduling
- **Financial Operations**: Create invoices and estimates with line items
- **Communication**: Initiate calls, texts, emails, and navigation
- **Navigation Support**: Return navigation instructions for mobile apps
- **Multi-language Support**: Text-to-speech in multiple languages

## Installation

```bash
npm install field-service-agent field-service-sdk
```

## Quick Start

```typescript
import { FieldServiceAgent } from 'field-service-agent';
import { FieldServiceSDK } from 'field-service-sdk';

// First, initialize the SDK
const sdk = new FieldServiceSDK({
  baseUrl: 'https://api.fieldservice.com',
  token: 'your-auth-token'
});

// Then create the agent with the SDK instance
const agent = new FieldServiceAgent({
  sdk: sdk, // Pass the SDK instance
  language: 'en-US',
  timezone: 'America/New_York',
  enableDebugLogging: true,
  googlePlacesApiKey: 'your-google-places-key', // Optional
  onNavigationRequest: (action) => {
    // Handle navigation in your app
    console.log('Navigate to:', action.screen, action.params);
  },
  onCommunicationRequest: (action) => {
    // Handle communication actions in your app
    console.log('Communication:', action.method, action.recipient);
  }
});

// Initialize the agent
await agent.initialize();

// Process a text command
const response = await agent.processCommand('Create a new client named John Doe');
console.log(response);

// Process audio input directly
const audioResponse = await agent.processCommand({
  audioContent: 'base64_encoded_audio_data',
  inputEncoding: 'WEBM_OPUS', // optional, defaults to WEBM_OPUS
  inputSampleRate: 48000       // optional, defaults to 48000
});
console.log(audioResponse);
```

## Audio Input Support

The agent can process audio input directly without requiring separate transcription. This is useful for voice-enabled applications:

```typescript
// Process audio input
const response = await agent.processCommand({
  audioContent: audioBase64,      // Base64 encoded audio data
  inputEncoding: 'WEBM_OPUS',     // Audio encoding format
  inputSampleRate: 48000          // Sample rate in Hz
});

// The response includes both text and audio
if (response.audioContent) {
  // Play the audio response
  const audioBlob = base64ToBlob(response.audioContent, 'audio/mpeg');
  const audio = new Audio(URL.createObjectURL(audioBlob));
  audio.play();
}
```

### Supported Audio Formats

- **WEBM_OPUS**: Default format, 48kHz sample rate
- **M4A**: Alternative format if needed
- Other formats supported by the Vertex AI API

## Context Awareness

The agent can be made aware of what the user is currently viewing in your app. This enables more intuitive commands that operate on the current context.

### Setting Current Screen Context

```typescript
// When user navigates to a client details screen
agent.setCurrentScreen('Client', 'client-123');

// When user navigates to a job details screen
agent.setCurrentScreen('Job', 'job-456');

// When user navigates to an invoice
agent.setCurrentScreen('Invoice', 'invoice-789');

// When user navigates to an estimate
agent.setCurrentScreen('Estimate', 'estimate-012');

// Clear context when leaving detail screens
agent.setCurrentScreen(undefined, undefined);
```

### Context-Aware Commands

When context is set, users can issue commands that automatically apply to the current screen:

```typescript
// While viewing a client
agent.setCurrentScreen('Client', 'client-123');
await agent.processCommand('Create a new job'); // Creates job for client-123
await agent.processCommand('Update email to john@example.com'); // Updates client-123's email

// While viewing a job
agent.setCurrentScreen('Job', 'job-456');
await agent.processCommand('Create an invoice'); // Creates invoice for job-456
await agent.processCommand('Add estimate'); // Creates estimate for job-456
```

### Passing Context with Commands

You can also pass context directly with each command:

```typescript
const response = await agent.processCommand(
  'Create an invoice',
  {
    currentScreenType: 'Job',
    currentRecordId: 'job-789'
  }
);
```

## Conversation History & Follow-up Questions

The agent maintains conversation history within a session, enabling natural follow-up questions and contextual responses.

### Automatic Conversation Tracking

Every command and response is automatically tracked:

```typescript
// First command
const response1 = await agent.processCommand('Show me clients named John');
// Response: "I found 3 clients named John: John Doe, John Smith, John Williams. Which one would you like to see?"

// Follow-up command - the agent knows this is about the previous question
const response2 = await agent.processCommand('The second one');
// The agent understands "the second one" refers to John Smith from the previous response
```

### Managing Conversation History

```typescript
// Clear history to start a new conversation
agent.clearConversationHistory();

// Get the full conversation history
const history = agent.getConversationHistory();
console.log(history);
// [
//   { id: 'msg_123_user', text: 'Show me clients named John', isUser: true },
//   { id: 'msg_124_agent', text: 'I found 3 clients...', isUser: false },
//   { id: 'msg_125_user', text: 'The second one', isUser: true },
//   { id: 'msg_126_agent', text: 'Opening client details...', isUser: false }
// ]

// Get only recent messages
const recentMessages = agent.getRecentMessages(5);
```

### Handling Follow-up Questions

The agent automatically detects when it needs more information:

```typescript
const response = await agent.processCommand('Create an invoice for John Doe');

if (response.followUpQuestion) {
  // The agent is asking a follow-up question
  console.log(response.message); // "John Doe has multiple jobs. Which job is this invoice for?"
  
  // Show options to user and get their selection
  const followUp = await agent.processCommand('The plumbing repair job');
  // The agent will now create the invoice for the correct job
}
```

### Multi-turn Conversations

```typescript
// Complex multi-step task
await agent.processCommand('I need to create an invoice');
// "Which client is this invoice for?"

await agent.processCommand('John Doe');
// "John Doe has 2 jobs: Plumbing Repair and Kitchen Remodel. Which job is this invoice for?"

await agent.processCommand('The kitchen remodel');
// "What items should I add to the invoice?"

await agent.processCommand('Labor for 8 hours at $50 per hour');
// "I created invoice #123 for John Doe's Kitchen Remodel job with 1 item totaling $400.00."
```

## Configuration

The `FieldServiceAgent` constructor accepts the following configuration options:

| Option | Type | Required | Description |
|--------|------|----------|-------------|
| `authToken` | string | Yes | Authentication token for the Field Service API |
| `baseUrl` | string | Yes | Base URL for the Field Service API |
| `language` | string | No | Language for text-to-speech (default: 'en-US') |
| `timezone` | string | No | Timezone for scheduling (default: system timezone) |
| `enableDebugLogging` | boolean | No | Enable debug logging (default: false) |
| `onNavigationRequest` | function | No | Callback for navigation actions |
| `onCommunicationRequest` | function | No | Callback for communication actions |

## Usage Examples

### Creating a Client

```typescript
const response = await agent.processCommand(
  'Create a new client named Sarah Johnson with phone 555-1234'
);

if (response.success) {
  console.log('Client created:', response.result.data.client);
}
```

### Creating a Job

```typescript
// Update context to current client
agent.updateContext({
  currentClient: { id: 'client-123', name: 'Sarah Johnson' }
});

const response = await agent.processCommand(
  'Create a new job for carpet cleaning scheduled for tomorrow at 2pm'
);
```

### Creating an Invoice

```typescript
// Update context to current job
agent.updateContext({
  currentJob: { id: 'job-456', title: 'Carpet Cleaning' }
});

const response = await agent.processCommand(
  'Create an invoice for carpet cleaning $50 per room, 3 rooms'
);
```

### Handling Navigation

```typescript
const agent = new FieldServiceAgent({
  // ... other config
  onNavigationRequest: (action) => {
    switch (action.screen) {
      case 'job_details':
        navigation.navigate('JobDetailsScreen', { jobId: action.params.id });
        break;
      case 'client_details':
        navigation.navigate('ClientDetailsScreen', { clientId: action.params.id });
        break;
      // ... handle other screens
    }
  }
});

await agent.processCommand('Open job details for carpet cleaning');
```

### Handling Communications

```typescript
const agent = new FieldServiceAgent({
  // ... other config
  onCommunicationRequest: (action) => {
    switch (action.method) {
      case 'call':
        Linking.openURL(`tel:${action.recipient.phone}`);
        break;
      case 'sms':
        Linking.openURL(`sms:${action.recipient.phone}`);
        break;
      case 'email':
        const { subject, body, attachmentUrl } = action.content;
        // Handle email with attachment
        break;
      case 'navigation':
        // Open navigation app
        const address = action.recipient.address;
        const app = action.navigationApp; // 'waze', 'google_maps', 'apple_maps'
        // Handle navigation
        break;
    }
  }
});

await agent.processCommand('Call Sarah Johnson');
await agent.processCommand('Email the invoice to the client');
```

## Response Structure

All commands return an `AgentResponse` object:

```typescript
interface AgentResponse {
  success: boolean;
  message: string;
  functionCalled?: string;
  functionArgs?: any;
  result?: {
    success: boolean;
    message: string;
    data?: any;
    action?: NavigationAction | CommunicationAction;
    requiresUserAction?: boolean;
  };
  error?: string;
  debugInfo?: any; // Only when enableDebugLogging is true
}
```

## Action Types

### Navigation Actions

```typescript
interface NavigationAction {
  type: 'navigation';
  screen: 'job_details' | 'client_details' | 'invoice_details' | 
          'estimate_details' | 'calendar_daily' | 'calendar_weekly';
  params: {
    id?: string;
    // ... other params
  };
}
```

### Communication Actions

```typescript
interface CommunicationAction {
  type: 'communication';
  method: 'call' | 'sms' | 'email' | 'navigation';
  recipient: {
    id: string;
    name: string;
    phone?: string;
    email?: string;
    address?: string;
  };
  content?: {
    subject?: string;
    body?: string;
    attachmentUrl?: string;
  };
  navigationApp?: 'waze' | 'google_maps' | 'apple_maps';
}
```

## Context Management

Update the agent's context when the user navigates or selects items:

```typescript
// When user views a client
agent.updateContext({
  currentScreenType: 'Client',
  currentRecordId: 'client-123',
  currentClient: { id: 'client-123', name: 'Sarah Johnson' }
});

// When user views a job
agent.updateContext({
  currentScreenType: 'Job',
  currentRecordId: 'job-456',
  currentJob: { id: 'job-456', title: 'Carpet Cleaning', clientId: 'client-123' }
});
```

## Text-to-Speech

Generate audio for agent responses:

```typescript
const audioBuffer = await agent.getTextToSpeechAudio(
  'Invoice created successfully',
  'en-US'
);

// Play audio in your app
const audio = new Audio(URL.createObjectURL(new Blob([audioBuffer])));
await audio.play();
```

## Available Commands

The agent understands natural language commands for:

- **Clients**: "Create a new client...", "Update client phone number...", "Show client details"
- **Jobs**: "Create a job for...", "Schedule for tomorrow at 2pm", "Update job status"
- **Invoices**: "Create an invoice for...", "Email invoice to client", "List all invoices"
- **Estimates**: "Create an estimate for...", "Email estimate to client"
- **Communication**: "Call client", "Text client", "Email client", "Navigate to client"
- **Calendar**: "Show today's schedule", "Show weekly calendar"

## Error Handling

```typescript
try {
  const response = await agent.processCommand(command);
  
  if (!response.success) {
    console.error('Command failed:', response.message);
    if (response.error) {
      console.error('Error details:', response.error);
    }
  }
} catch (error) {
  console.error('Agent error:', error);
}
```

## API Reference

### Main Methods

#### `processCommand(command)`
Process a text or audio command.

**Parameters:**
- `command`: Can be either:
  - `string`: Text command to process
  - `object`: Audio input with:
    - `audioContent` (string): Base64 encoded audio data
    - `inputEncoding` (string, optional): Audio format (default: 'WEBM_OPUS')
    - `inputSampleRate` (number, optional): Sample rate in Hz (default: 48000)

**Returns:** `Promise<AgentResponse>` with:
- `success` (boolean): Whether the command was processed successfully
- `message` (string): Response message
- `audioContent` (string, optional): Base64 encoded audio response
- `functionCalled` (string, optional): Name of function executed
- `functionArgs` (any, optional): Arguments passed to function

#### `processVoiceCommand(command, context?)`
Process a text command with simplified response for TTS. Use `processCommand` with audio input for direct audio processing.

#### `processVoiceInput(audioContent, inputEncoding?, inputSampleRate?, context?)`
Process audio input directly. This method is now deprecated - use `processCommand` with an audio object instead.

#### `setCurrentScreen(screenType?, recordId?)`
Set the current context for the agent.

#### `setLanguage(language)`
Set the agent's response language (e.g., 'en-US', 'es-ES', 'fr-FR').

#### `setVoiceName(voiceName)`
Set the TTS voice name for audio responses.

## TypeScript Support

This package includes full TypeScript definitions. Import types as needed:

```typescript
import { 
  FieldServiceAgent,
  AgentResponse,
  NavigationAction,
  CommunicationAction,
  FieldServiceAgentConfig
} from 'field-service-agent';
```

## License

MIT

## Support

For issues and feature requests, please visit the [GitHub repository](https://github.com/field-service/agent-package).