@sammy-labs/sammy-three/README.md

Sammy Agent Core: AI voice agent with screen capture, memory management, and live API integration

# Sammy Agent Core

Core AI voice agent functionality with screen capture, memory management, and live API integration.

## Installation

```bash
npm install @sammy-labs/sammy-three
```

For UI components, also install:

```bash
npm install @sammy-labs/sammy-three-ui-kit
```

## Features

- 🎙️ **Voice Interaction** - Real-time voice communication with AI agents
- 📱 **Screen Capture** - Automatic screen recording and context sharing
- 🧠 **Memory Management** - Persistent conversation memory and context
- 🔌 **Service Abstraction** - Pluggable backend services
- ⚡ **Live API** - WebSocket-based real-time communication
- 🎛️ **React Hooks** - Simple integration with React applications

## Quick Start

```tsx
import {
  SammyAgentProvider,
  createDefaultServices,
  useSammyAgent,
} from '@sammy-labs/sammy-three';

// Set up services (points to your backend)
const services = createDefaultServices('/api');

function App() {
  return (
    <SammyAgentProvider
      services={services}
      onTokenExpired={() => handleTokenRefresh()}
      onError={(error) => console.error('Agent error:', error)}
    >
      <AgentInterface />
    </SammyAgentProvider>
  );
}

function AgentInterface() {
  const { isConnected, connect, disconnect, sendMessage } = useSammyAgent({
    agentMode: 'user',
    autoConnect: true,
    onReady: () => console.log('Agent connected!'),
  });

  return (
    <div>
      <div>Status: {isConnected ? 'Connected' : 'Disconnected'}</div>
      <button onClick={connect}>Connect</button>
      <button onClick={disconnect}>Disconnect</button>
      <button onClick={() => sendMessage('Hello!')}>Send Message</button>
    </div>
  );
}
```

## Architecture

### Provider + Hook Pattern

The package follows a clean Provider + Hook architecture:

- **`SammyAgentProvider`** - Manages global agent state and connections
- **`useSammyAgent`** - Simple hook for component integration
- **Service Layer** - Abstracts external dependencies (APIs, storage, etc.)

### Core Classes

- **`SammyAgentCore`** - Main domain logic class
- **`ScreenCaptureManager`** - Handles screen/video capture
- **`MemoryManager`** - Manages conversation memory
- **`ToolManager`** - Processes agent tool calls

## API Reference

### SammyAgentProvider

```tsx
interface SammyAgentProviderProps {
  services: SammyAgentServices;
  config?: Partial<SammyAgentConfig>;
  onTokenExpired?: () => void;
  onError?: (error: unknown) => void;
  onTurnComplete?: (summary: { user: string; agent: string }) => void;
  onMemoryUpdate?: (result: any) => void;
  onToolCall?: (tool: LiveServerToolCall) => void;
  onConnectionStateChange?: (connected: boolean) => void;
}
```

### useSammyAgent Hook

```tsx
interface UseSammyAgentOptions {
  autoConnect?: boolean;
  agentMode?: 'admin' | 'user';
  flowId?: string;
  isInternal?: boolean;
  onReady?: () => void;
  onError?: (error: Error) => void;
}

interface UseSammyAgentResult {
  // State
  isConnected: boolean;
  isConnecting: boolean;
  error: Error | null;

  // Actions
  connect: () => Promise<boolean>;
  disconnect: () => void;
  sendMessage: (message: string) => void;
  setMuted: (muted: boolean) => void;

  // Session info
  conversationId?: string;
  sessionId?: string;
}
```

### Service Interface

```tsx
interface SammyAgentServices {
  // Authentication
  getEphemeralToken(
    constraints: EphemeralTokenConstraints
  ): Promise<EphemeralTokenResponse>;

  // Knowledge & RAG
  fetchDocs(
    query: string,
    limit?: number,
    options?: any
  ): Promise<DocumentationResult[]>;
  batchFetchDocs?(
    queries: string[],
    options?: any
  ): Promise<DocumentationResult[]>;

  // Memory Management
  processMemories(args: ProcessMemoriesArgs): Promise<MemoryProcessResult>;
  searchMemories(options: SearchMemoriesOptions): Promise<MemorySearchResult>;

  // Message Handling
  sendMessages(payload: MessagePayload): Promise<void>;
  transcribeAudio?(payload: TranscriptionPayload): Promise<{ text: string }>;

  // Feedback
  submitFeedback?(feedback: FeedbackSubmission): Promise<void>;
  fetchFeedback?(query?: FeedbackQuery): Promise<FeedbackResult[]>;
  deleteFeedback?(feedbackId: string): Promise<void>;

  // System
  getSystemPrompt?(query: SystemPromptQuery): Promise<{ prompt: string }>;
  fetchConversations?(): Promise<ConversationData[]>;
  fetchMessages?(conversationId: string): Promise<MessageData[]>;
}
```

## Configuration

```tsx
interface SammyAgentConfig {
  captureMethod?: 'render' | 'screen' | 'video';
  debugLogs?: boolean;
  toolUse?: boolean;
  model?: string;
  captureConfig?: {
    frameRate?: number;
    quality?: number;
  };
}
```

## Build Modes & Console Logging

The package automatically manages console logging based on the build environment:

### Development Builds (Console logs retained)
```bash
npm run build       # 🔧 Console logs retained (development build)
npm run dev         # 🔧 Console logs retained (development build)
```

### Production Builds (Console logs dropped)
```bash
NODE_ENV=production npm run build    # 🗑️ Console logs dropped (production build)  
npm run build:prod                   # 🗑️ Console logs dropped (production build)
```

### Build Output
During build time, you'll see clear indicators of console log behavior:
- **Workers**: `🔧 Worker console logs will be retained` or `🗑️ Worker console logs will be dropped`
- **Main Package**: `🔧 Console logs retained` or `🗑️ Console logs dropped`
- **Summary**: `🔧 All worker console logs have been retained` or `🗑️ All worker console logs have been dropped`

This ensures optimal performance in production while preserving debug capabilities during development.

## Service Implementation

### Default Services

```tsx
import { createDefaultServices } from '@sammy-labs/sammy-three';

// Uses your existing API routes
const services = createDefaultServices('/api');
```

### Custom Services

```tsx
const customServices: SammyAgentServices = {
  getEphemeralToken: async (constraints) => {
    // Your custom token logic
    return { token: 'your-token' };
  },

  fetchDocs: async (query, limit) => {
    // Your custom documentation fetching
    return await myDocService.search(query, limit);
  },

  processMemories: async (args) => {
    // Your custom memory processing
    return await myMemoryService.process(args);
  },

  // ... implement other required methods
};
```

## Advanced Usage

### Screen Capture

```tsx
const { connect } = useSammyAgent({
  agentMode: 'user',
  // Configure screen capture
});

// The agent will automatically capture screen context
// when connected based on your configuration
```

### Memory Integration

```tsx
// Memory is automatically managed, but you can listen for updates
<SammyAgentProvider
  services={services}
  onMemoryUpdate={(result) => {
    console.log('Memory updated:', result);
    // Handle memory updates in your UI
  }}
>
```

### Tool Calls

```tsx
// Listen for tool calls from the agent
<SammyAgentProvider
  services={services}
  onToolCall={(tool) => {
    console.log('Agent called tool:', tool.functionCalls);
    // Handle agent tool calls
  }}
>
```

### Multiple Agents

```tsx
function MultiAgentApp() {
  return (
    <div>
      <SammyAgentProvider services={userServices}>
        <UserAgent />
      </SammyAgentProvider>

      <SammyAgentProvider services={adminServices}>
        <AdminAgent />
      </SammyAgentProvider>
    </div>
  );
}
```

## TypeScript Support

Full TypeScript support is included with comprehensive type definitions.