contaigents/AUDIO_GENERATION_TOOL_SPEC.md

Modular AI Content Ecosystem with Audio Generation

# Audio Generation Agent Tool Specification

## Overview

This specification defines an `AudioGenerationTool` that integrates the existing `AudioGenerator` utility into the agent tool architecture. The tool enables AI assistants to generate audio files from text content using various AI providers (starting with Gemini) and automatically store the generated audio files in the project workspace.

## Tool Architecture Integration

### Tool Class: `AudioGenerationTool`

The tool extends `BaseTool` and provides a structured interface for audio generation within the agent conversation flow.

#### Tool Definition
- **Name**: `audio_generation`
- **Description**: Generate audio files from text using AI providers (Gemini TTS)
- **Category**: Content Generation

#### Parameters

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `text` | string | Yes | - | Text content to convert to audio |
| `output_path` | string | Yes | - | Output file path (relative to project root) |
| `provider` | string | No | "gemini" | AI provider for audio generation |
| `voice` | string | No | "zephyr" | Voice to use for generation |
| `model` | string | No | - | Specific model to use (provider-dependent) |

#### Tool Call XML Format

```xml
<tool_call name="audio_generation" id="1">
  <text>Hello, this is a test of the audio generation feature.</text>
  <output_path>audio/greeting.wav</output_path>
  <voice>zephyr</voice>
  <provider>gemini</provider>
</tool_call>
```

#### Tool Response XML Format

**Successful Generation:**
```xml
<tool_response id="1">
  <tool_call name="audio_generation">
    <text>Hello, this is a test of the audio generation feature.</text>
    <output_path>audio/greeting.wav</output_path>
    <voice>Breeze</voice>
    <provider>gemini</provider>
  </tool_call>
  <result>
    <success>true</success>
    <message>Audio generated successfully</message>
    <data>
      <output_file_path>audio/greeting.wav</output_file_path>
      <file_size>45632</file_size>
      <duration_estimate>3.2s</duration_estimate>
      <provider>gemini</provider>
      <voice>Breeze</voice>
      <text_length>47</text_length>
    </data>
  </result>
</tool_response>
```

**Error Response:**
```xml
<tool_response id="1">
  <tool_call name="audio_generation">
    <text>Hello world</text>
    <output_path>audio/test.wav</output_path>
  </tool_call>
  <result>
    <success>false</success>
    <error>No configured gemini provider found. Please configure the gemini provider using 'contaigents configure' command.</error>
  </result>
</tool_response>
```

## LLM Configuration Integration

The AudioGenerationTool integrates with the existing LLM configuration system instead of handling API keys separately. This provides several benefits:

### Configuration Management
- **Unified Configuration**: Uses the same configuration system as other LLM providers
- **Environment Variables**: Supports `env:VARIABLE_NAME` syntax for secure API key storage
- **Provider Selection**: Automatically maps audio providers to configured LLM providers
- **No Duplicate Keys**: Eliminates the need to store API keys in multiple places

### Provider Mapping
- `gemini` → `Google` LLM provider
- `openai` → `OpenAI` LLM provider
- Other providers follow the same pattern

### Setup Requirements
Before using the audio generation tool, ensure you have a configured LLM provider:

```bash
# Configure Google provider for Gemini audio generation
contaigents configure
# Select "Google" and provide your API key
```

Or manually create the configuration file:
```json
// .config/llm_config/google.json
{
  "apiKey": "env:GEMINI_API_KEY",
  "model": "gemini-2.5-flash"
}
```

## Implementation Details

### AudioGenerationTool Class Structure

```typescript
export class AudioGenerationTool extends BaseTool {
  private baseDir: string;
  
  constructor(baseDir?: string) {
    this.baseDir = baseDir || process.cwd();
  }
  
  getName(): string {
    return 'audio_generation';
  }
  
  getDescription(): string {
    return 'Generate audio files from text using AI providers (Gemini TTS). Supports various voices and automatically saves files to specified paths.';
  }
  
  getParameters(): ToolParameter[] {
    // Parameter definitions as specified above
  }
  
  async execute(parameters: Record<string, any>): Promise<ToolResult> {
    // Implementation using AudioGenerator
  }
}
```

### Integration with AudioGenerator

The tool will:
1. Validate input parameters
2. Resolve API key from parameter or environment variables
3. Create `AudioGenerator` instance with provided configuration
4. Generate audio using `generateAudio()` method
5. Save audio file using `saveAudioToFile()` method
6. Return structured response with file information

### File Path Handling

- **Relative Paths**: Resolved relative to project root (`baseDir`)
- **Directory Creation**: Automatically create parent directories if they don't exist
- **File Extension**: Automatically determined by AudioGenerator based on provider output
- **Path Validation**: Ensure output path is within project boundaries for security

### Error Handling

- **Provider Not Configured**: Clear error message with guidance to use `contaigents configure`
- **Invalid Voice**: List available voices for the provider
- **File System Errors**: Directory creation, permission issues
- **Provider Errors**: Network issues, API limits, invalid requests
- **Text Validation**: Empty text, text too long for provider limits

## Usage Examples

### Basic Audio Generation
```xml
<tool_call name="audio_generation" id="1">
  <text>Welcome to our podcast. Today we'll discuss the future of AI.</text>
  <output_path>podcast/intro.wav</output_path>
</tool_call>
```

### Custom Voice and Provider
```xml
<tool_call name="audio_generation" id="2">
  <text>This is a test with a different voice.</text>
  <output_path>tests/voice_test.wav</output_path>
  <voice>despina</voice>
  <provider>gemini</provider>
</tool_call>
```

### Long Content Generation
```xml
<tool_call name="audio_generation" id="3">
  <text><![CDATA[
This is a longer piece of content that might span multiple paragraphs.

It includes various punctuation marks, numbers like 123, and special characters.
The AudioGenerator will handle chunking this content appropriately for the provider's limits.
]]></text>
  <output_path>content/long_narration.wav</output_path>
  <voice>puck</voice>
</tool_call>
```

## Agent Conversation Flow

### Example Interaction

**User:** "Can you create an audio version of the README file for accessibility?"

**Agent Response:**
```
I'll read the README file and generate an audio version for you.

<tool_call name="read_file" id="1">
  <file_path>README.md</file_path>
</tool_call>

<tool_call name="audio_generation" id="2">
  <text>[README content will be inserted here after reading]</text>
  <output_path>audio/readme_narration.wav</output_path>
  <voice>zephyr</voice>
</tool_call>
```

**Tool Responses:**
```xml
<tool_response id="1">
  <tool_call name="read_file">
    <file_path>README.md</file_path>
  </tool_call>
  <result>
    <success>true</success>
    <data>
      <content># Project Title
This is the project description...</content>
    </data>
  </result>
</tool_response>

<tool_response id="2">
  <tool_call name="audio_generation">
    <text># Project Title
This is the project description...</text>
    <output_path>audio/readme_narration.wav</output_path>
    <voice>zephyr</voice>
  </tool_call>
  <result>
    <success>true</success>
    <message>Audio generated successfully</message>
    <data>
      <output_file_path>audio/readme_narration.wav</output_file_path>
      <file_size>156789</file_size>
      <duration_estimate>12.4s</duration_estimate>
    </data>
  </result>
</tool_response>
```

**Final Agent Message:**
"I've successfully created an audio version of your README file! The narration has been saved to `audio/readme_narration.wav` (156KB, approximately 12.4 seconds). You can now use this for accessibility purposes or to listen to your project documentation."

## Integration Points

### ToolManager Registration
The `AudioGenerationTool` will be registered in `ToolManager.initializeTools()` alongside existing tools.

### Environment Configuration
- Supports `GEMINI_API_KEY` environment variable
- Falls back to parameter-provided API key
- Future: Integration with `contaigents configure` command

### File System Integration
- Uses existing `FileService` patterns for path resolution
- Integrates with project file tree context
- Respects project boundaries for security

## Future Enhancements

### Additional Providers
- OpenAI TTS integration
- Azure Speech Services
- AWS Polly support

### Advanced Features
- Audio format options (MP3, WAV, OGG)
- Speed/pitch control parameters
- SSML support for advanced speech control
- Batch processing for multiple files

### Shared Memory Integration
- Store large audio files in shared memory system
- Reference audio files by memory IDs
- Enable audio processing workflows

## Testing Strategy

### Unit Tests
- Parameter validation
- AudioGenerator integration
- File path resolution
- Error handling scenarios

### Integration Tests
- End-to-end tool execution
- XML parsing and response formatting
- File system operations
- Provider API integration (with real keys)

### Agent Conversation Tests
- Multi-tool workflows (read file → generate audio)
- Error recovery scenarios
- Long content handling