contaigents/AUDIO_GENERATION_TOOL_SPEC.md

Modular AI Content Ecosystem with Audio Generation

# Extended Audio Generation Tool Specification

## Overview

This specification defines an enhanced `AudioGenerationTool` that supports both **speech generation** (TTS) and **music generation** capabilities. The tool integrates multiple AI providers and content types, enabling AI assistants to generate speech, background music, sound effects, and other audio content for use in complex audio projects.

## Content Types Supported

### 1. Speech Generation (Text-to-Speech)
- **Provider**: Gemini TTS
- **Use Cases**: Narration, dialogue, announcements
- **Input**: Text content
- **Output**: High-quality speech audio

### 2. Music Generation
- **Provider**: Lyria 2 (Google/Vertex AI)
- **Use Cases**: Background music, intros, outros, ambient sounds
- **Input**: Text descriptions of musical style/mood
- **Output**: High-quality instrumental music tracks (30 seconds, 48kHz WAV)

### 3. Sound Effects (Future)
- **Provider**: AudioGen or similar
- **Use Cases**: Ambient sounds, transitions, effects
- **Input**: Text descriptions of sounds
- **Output**: Sound effect audio

## Tool Architecture Integration

### Tool Class: `AudioGenerationTool`

The tool extends `BaseTool` and provides a structured interface for audio generation within the agent conversation flow.

#### Tool Definition
- **Name**: `audio_generation`
- **Description**: Generate audio files from text using AI providers (Gemini TTS)
- **Category**: Content Generation

#### Enhanced Parameters

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `text` | string | Yes | - | Text content or music description |
| `output_path` | string | Yes | - | Output file path (relative to project root) |
| `content_type` | string | No | "speech" | Type of content: speech, music, sound_effect |
| `provider` | string | No | "auto" | AI provider (auto-selected based on content_type) |
| `voice` | string | No | "zephyr" | Voice for speech generation |
| `style` | string | No | - | Music style/genre for music generation |
| `duration` | number | No | 30 | Duration in seconds for music generation |
| `model` | string | No | - | Specific model to use (provider-dependent) |

#### Tool Call XML Format Examples

**Speech Generation:**
```xml
<tool_call name="audio_generation" id="1">
  <text>Welcome to our podcast about AI technology</text>
  <output_path>audio/intro_speech.wav</output_path>
  <content_type>speech</content_type>
  <voice>zephyr</voice>
</tool_call>
```

**Music Generation:**
```xml
<tool_call name="audio_generation" id="2">
  <text>Upbeat electronic background music for a tech podcast</text>
  <output_path>audio/background_music.wav</output_path>
  <content_type>music</content_type>
  <style>electronic</style>
  <duration>60</duration>
</tool_call>
```

**Sound Effect Generation:**
```xml
<tool_call name="audio_generation" id="3">
  <text>Gentle rain sounds for relaxation</text>
  <output_path>audio/rain_ambience.wav</output_path>
  <content_type>sound_effect</content_type>
  <duration>120</duration>
</tool_call>
```

#### Tool Response XML Format

**Successful Generation:**
```xml
<tool_response id="1">
  <tool_call name="audio_generation">
    <text>Hello, this is a test of the audio generation feature.</text>
    <output_path>audio/greeting.wav</output_path>
    <voice>Breeze</voice>
    <provider>gemini</provider>
  </tool_call>
  <result>
    <success>true</success>
    <message>Audio generated successfully</message>
    <data>
      <output_file_path>audio/greeting.wav</output_file_path>
      <file_size>45632</file_size>
      <duration_estimate>3.2s</duration_estimate>
      <provider>gemini</provider>
      <voice>Breeze</voice>
      <text_length>47</text_length>
    </data>
  </result>
</tool_response>
```

**Error Response:**
```xml
<tool_response id="1">
  <tool_call name="audio_generation">
    <text>Hello world</text>
    <output_path>audio/test.wav</output_path>
  </tool_call>
  <result>
    <success>false</success>
    <error>No configured gemini provider found. Please configure the gemini provider using 'contaigents configure' command.</error>
  </result>
</tool_response>
```

## LLM Configuration Integration

The AudioGenerationTool integrates with the existing LLM configuration system instead of handling API keys separately. This provides several benefits:

### Configuration Management
- **Unified Configuration**: Uses the same configuration system as other LLM providers
- **Environment Variables**: Supports `env:VARIABLE_NAME` syntax for secure API key storage
- **Provider Selection**: Automatically maps audio providers to configured LLM providers
- **No Duplicate Keys**: Eliminates the need to store API keys in multiple places

### Provider Mapping
- `gemini` → `Google` LLM provider
- `openai` → `OpenAI` LLM provider
- Other providers follow the same pattern

### Setup Requirements
Before using the audio generation tool, ensure you have a configured LLM provider:

```bash
# Configure Google provider for Gemini audio generation
contaigents configure
# Select "Google" and provide your API key
```

Or manually create the configuration file:
```json
// .config/llm_config/google.json
{
  "apiKey": "env:GEMINI_API_KEY",
  "model": "gemini-2.5-flash"
}
```

## Implementation Plan

### Phase 1: Enhanced AudioGenerator Architecture

```typescript
export class AudioGenerator {
  private provider: string;
  private contentType: 'speech' | 'music' | 'sound_effect';
  private apiKey: string;

  constructor(config: {
    provider?: string;
    contentType?: string;
    apiKey: string;
    // ... other config
  }) {
    this.contentType = config.contentType || 'speech';
    this.provider = this.selectProvider(config.provider, this.contentType);
  }

  private selectProvider(requested?: string, contentType?: string): string {
    if (requested && requested !== 'auto') return requested;

    // Auto-select provider based on content type
    switch (contentType) {
      case 'speech': return 'gemini';
      case 'music': return 'musicgen';
      case 'sound_effect': return 'audiogen';
      default: return 'gemini';
    }
  }

  async generateAudio(prompt: string, options?: any): Promise<Buffer> {
    switch (this.contentType) {
      case 'speech':
        return this._generateSpeechWithGemini(prompt);
      case 'music':
        return this._generateMusicWithMusicGen(prompt, options);
      case 'sound_effect':
        return this._generateSoundWithAudioGen(prompt, options);
      default:
        throw new Error(`Unsupported content type: ${this.contentType}`);
    }
  }
}
```

### Phase 2: Hybrid Music Generation Implementation

#### Smart Provider Selection
```typescript
private async generateMusic(
  prompt: string,
  options: {
    duration?: number;
    style?: string;
    negative_prompt?: string;
    seed?: number;
    prefer_local?: boolean;
  } = {}
): Promise<{ success: boolean; data?: Buffer; error?: string; provider_used?: string }> {

  // Check user preference or try cloud first
  if (!options.prefer_local) {
    try {
      console.log('🌐 Attempting music generation with Lyria 2...');
      const result = await this._generateMusicWithLyria(prompt, options);
      return { success: true, data: result, provider_used: 'lyria-2' };
    } catch (error) {
      console.log(`⚠️ Lyria 2 failed: ${error.message}`);
      console.log('🔄 Falling back to local MusicGen...');
    }
  }

  // Try local MusicGen
  try {
    const result = await this._generateMusicWithMusicGen(prompt, options);
    return { success: true, data: result, provider_used: 'musicgen-local' };
  } catch (error) {
    console.log(`⚠️ Local MusicGen failed: ${error.message}`);

    // Check if it's a connection error (Docker not running)
    if (error.message.includes('ECONNREFUSED') || error.message.includes('connection refused')) {
      return await this._handleMusicGenSetup(prompt, options);
    }

    return { success: false, error: `Both music providers failed: ${error.message}` };
  }
}
```

#### Lyria 2 Implementation
```typescript
private async _generateMusicWithLyria(
  prompt: string,
  options: {
    duration?: number;
    style?: string;
    negative_prompt?: string;
    seed?: number;
  } = {}
): Promise<Buffer> {
  const {
    duration = 30, // Lyria 2 generates 30-second clips
    style,
    negative_prompt,
    seed
  } = options;

  // Enhance prompt with style if provided
  const enhancedPrompt = style ? `${style}: ${prompt}` : prompt;

  // Call Lyria 2 via Vertex AI (same client as Gemini)
  const response = await this.vertexAIClient.predict({
    endpoint: `projects/${this.projectId}/locations/${this.location}/publishers/google/models/lyria-002:predict`,
    instances: [{
      prompt: enhancedPrompt,
      negative_prompt,
      seed
    }],
    parameters: {}
  });

  // Decode base64 audio content
  const audioContent = response.predictions[0].audioContent;
  return Buffer.from(audioContent, 'base64');
}
```

#### Local MusicGen Implementation
```typescript
private async _generateMusicWithMusicGen(
  prompt: string,
  options: {
    duration?: number;
    style?: string;
  } = {}
): Promise<Buffer> {
  const {
    duration = 30,
    style
  } = options;

  // Enhance prompt with style if provided
  const enhancedPrompt = style ? `${style}: ${prompt}` : prompt;

  // Call local MusicGen server
  const response = await fetch('http://localhost:5000/generate', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({
      prompt: enhancedPrompt,
      duration
    })
  });

  if (!response.ok) {
    throw new Error(`MusicGen server error: ${response.statusText}`);
  }

  const result = await response.json();
  return Buffer.from(result.audio, 'base64');
}
```

### Phase 3: Provider Integration Options

#### Hybrid Approach: Lyria 2 + Local MusicGen Fallback

**Primary: Lyria 2 via Vertex AI**
- **Setup**: Uses existing Gemini API configuration
- **Pros**:
  - No additional setup required
  - Same authentication as existing speech generation
  - High-quality output (48kHz WAV)
  - No local infrastructure needed
  - Integrated with Google Cloud ecosystem
- **Cons**: API costs ($0.06 per 30-second clip)
- **When to use**: Default choice, always try first

**Fallback: Local MusicGen with Auto-Setup**
- **Setup**: Automatic Docker container deployment via CommandExecutionTool
- **Pros**:
  - No per-request costs after setup
  - Works offline
  - Full control over generation
  - Automatic setup with user consent
- **Cons**: Requires Docker, GPU recommended, initial setup time
- **When to use**:
  - Lyria 2 unavailable (API errors, quota exceeded)
  - User preference for local generation
  - High-volume usage scenarios

**Intelligent Provider Selection Logic:**
1. **Try Lyria 2 first** (cloud-first approach)
2. **On failure**: Check if local MusicGen is running
3. **If not running**: Ask user permission to set up Docker container
4. **Auto-setup**: Use CommandExecutionTool to run Docker commands
5. **Retry**: Generate music with local MusicGen
6. **Cache preference**: Remember user's choice for future requests
```

### Integration with AudioGenerator

The tool will:
1. Validate input parameters
2. Resolve API key from parameter or environment variables
3. Create `AudioGenerator` instance with provided configuration
4. Generate audio using `generateAudio()` method
5. Save audio file using `saveAudioToFile()` method
6. Return structured response with file information

### File Path Handling

- **Relative Paths**: Resolved relative to project root (`baseDir`)
- **Directory Creation**: Automatically create parent directories if they don't exist
- **File Extension**: Automatically determined by AudioGenerator based on provider output
- **Path Validation**: Ensure output path is within project boundaries for security

### Error Handling

- **Provider Not Configured**: Clear error message with guidance to use `contaigents configure`
- **Invalid Voice**: List available voices for the provider
- **File System Errors**: Directory creation, permission issues
- **Provider Errors**: Network issues, API limits, invalid requests
- **Text Validation**: Empty text, text too long for provider limits

## Practical Implementation Steps

### Step 1: Extend AudioGenerationTool Parameters
```typescript
getParameters(): ToolParameter[] {
  return [
    // Existing parameters...
    {
      name: 'content_type',
      type: 'string',
      description: 'Type of audio content: speech, music, sound_effect',
      required: false,
      default: 'speech'
    },
    {
      name: 'style',
      type: 'string',
      description: 'Music style/genre (for music generation)',
      required: false
    },
    {
      name: 'duration',
      type: 'number',
      description: 'Duration in seconds (for music/sound generation)',
      required: false,
      default: 30
    }
  ];
}
```

### Step 2: Automatic Docker Setup with User Consent
```typescript
private async _handleMusicGenSetup(
  prompt: string,
  options: any
): Promise<{ success: boolean; data?: Buffer; error?: string; provider_used?: string }> {

  // Ask user for permission to set up local MusicGen
  const userConsent = await this._requestUserConsent();
  if (!userConsent) {
    return {
      success: false,
      error: 'Local MusicGen setup declined by user. Please use Lyria 2 or set up MusicGen manually.'
    };
  }

  try {
    console.log('🐳 Setting up local MusicGen with Docker...');

    // Check if Docker is available
    await this._checkDockerAvailability();

    // Set up MusicGen container
    await this._setupMusicGenContainer();

    // Wait for container to be ready
    await this._waitForMusicGenReady();

    // Retry music generation
    console.log('✅ MusicGen setup complete. Retrying music generation...');
    const result = await this._generateMusicWithMusicGen(prompt, options);
    return { success: true, data: result, provider_used: 'musicgen-local' };

  } catch (error) {
    return {
      success: false,
      error: `Failed to set up local MusicGen: ${error.message}`
    };
  }
}

private async _requestUserConsent(): Promise<boolean> {
  // This would integrate with the chat system to ask the user
  console.log(`
🎵 MUSIC GENERATION SETUP REQUIRED

Lyria 2 is currently unavailable. Would you like to set up local music generation?

This will:
- Download and run a Docker container (~2GB)
- Require Docker to be installed on your system
- Enable offline music generation
- Take 2-3 minutes for initial setup

Continue with local setup? (y/n)
  `);

  // In a real implementation, this would use the chat system
  // For now, return true for automatic setup
  return true;
}

private async _checkDockerAvailability(): Promise<void> {
  try {
    // Use CommandExecutionTool to check Docker
    const result = await this.commandTool.execute({
      command: 'docker --version',
      timeout: 10000
    });

    if (!result.success) {
      throw new Error('Docker is not installed. Please install Docker first.');
    }

    console.log('✅ Docker is available');
  } catch (error) {
    throw new Error(`Docker check failed: ${error.message}`);
  }
}

private async _setupMusicGenContainer(): Promise<void> {
  const commands = [
    // Pull the MusicGen image (or build if custom)
    'docker pull pytorch/pytorch:2.0.1-cuda11.7-cudnn8-runtime',

    // Create Dockerfile content
    `cat > /tmp/musicgen-dockerfile << 'EOF'
FROM pytorch/pytorch:2.0.1-cuda11.7-cudnn8-runtime

RUN pip install audiocraft transformers flask torch torchaudio

COPY musicgen_server.py /app/
WORKDIR /app

EXPOSE 5000
CMD ["python", "musicgen_server.py"]
EOF`,

    // Create the server script
    `cat > /tmp/musicgen_server.py << 'EOF'
from flask import Flask, request, jsonify
from audiocraft.models import MusicGen
import torch
import io
import base64
import torchaudio

app = Flask(__name__)
print("Loading MusicGen model...")
model = MusicGen.get_pretrained('facebook/musicgen-small')
print("Model loaded successfully!")

@app.route('/health', methods=['GET'])
def health():
    return jsonify({'status': 'ready'})

@app.route('/generate', methods=['POST'])
def generate_music():
    try:
        data = request.json
        prompt = data['prompt']
        duration = data.get('duration', 30)

        print(f"Generating music for: {prompt}")
        model.set_generation_params(duration=duration)
        wav = model.generate([prompt])

        # Convert to WAV format
        buffer = io.BytesIO()
        torchaudio.save(buffer, wav[0].cpu(), model.sample_rate, format='wav')
        audio_b64 = base64.b64encode(buffer.getvalue()).decode()

        return jsonify({
            'audio': audio_b64,
            'sample_rate': model.sample_rate,
            'duration': duration
        })
    except Exception as e:
        return jsonify({'error': str(e)}), 500

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)
EOF`,

    // Build the container
    'docker build -t musicgen-server -f /tmp/musicgen-dockerfile /tmp/',

    // Run the container
    'docker run -d --name musicgen-container -p 5000:5000 --gpus all musicgen-server'
  ];

  for (const command of commands) {
    console.log(`🔧 Running: ${command.split('\n')[0]}...`);
    const result = await this.commandTool.execute({
      command,
      timeout: 300000 // 5 minutes for model download
    });

    if (!result.success) {
      throw new Error(`Setup command failed: ${command}\nError: ${result.error}`);
    }
  }
}

private async _waitForMusicGenReady(): Promise<void> {
  console.log('⏳ Waiting for MusicGen to be ready...');

  const maxAttempts = 30; // 5 minutes max
  const delayMs = 10000; // 10 seconds between attempts

  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      const response = await fetch('http://localhost:5000/health');
      if (response.ok) {
        console.log('✅ MusicGen is ready!');
        return;
      }
    } catch (error) {
      // Container not ready yet
    }

    console.log(`⏳ Attempt ${attempt}/${maxAttempts} - MusicGen not ready yet...`);
    await new Promise(resolve => setTimeout(resolve, delayMs));
  }

  throw new Error('MusicGen container failed to start within timeout period');
}
```

### Step 3: Enhanced AudioGenerationTool Parameters
```typescript
// Add new parameters to existing tool
{
  name: 'negative_prompt',
  type: 'string',
  description: 'Elements to avoid in music generation (for music content_type)',
  required: false
},
{
  name: 'seed',
  type: 'number',
  description: 'Seed for reproducible music generation (for music content_type)',
  required: false
},
{
  name: 'prefer_local',
  type: 'boolean',
  description: 'Prefer local MusicGen over cloud Lyria 2 (for music content_type)',
  required: false,
  default: false
}
```

## Enhanced Usage Examples

### Complete Podcast Creation Workflow
```xml
<!-- Generate intro speech -->
<tool_call name="audio_generation" id="1">
  <text>Welcome to Tech Talk, your weekly dive into the world of technology</text>
  <output_path>audio/intro_speech.wav</output_path>
  <content_type>speech</content_type>
  <voice>zephyr</voice>
</tool_call>

<!-- Generate background music -->
<tool_call name="audio_generation" id="2">
  <text>Upbeat electronic background music for technology podcast</text>
  <output_path>audio/intro_music.wav</output_path>
  <content_type>music</content_type>
  <style>electronic</style>
  <duration>45</duration>
</tool_call>

<!-- Generate main content -->
<tool_call name="audio_generation" id="3">
  <text>Today we're discussing the latest developments in artificial intelligence</text>
  <output_path>audio/main_content.wav</output_path>
  <content_type>speech</content_type>
  <voice>despina</voice>
</tool_call>

<!-- Create project with all elements -->
<tool_call name="audio_project" id="4">
  <operation>create</operation>
  <project_path>projects/tech_talk_episode.json</project_path>
  <title>Tech Talk Episode 1</title>
  <clips>[
    {"file_path": "audio/intro_music.wav", "start_time": 0, "volume": 0.3},
    {"file_path": "audio/intro_speech.wav", "start_time": 2, "volume": 1.0},
    {"file_path": "audio/main_content.wav", "start_time": 15, "volume": 0.9}
  ]</clips>
</tool_call>
```

### Music-Only Generation with Fallback
```xml
<tool_call name="audio_generation" id="1">
  <text>Relaxing ambient music for meditation and focus</text>
  <output_path>music/ambient_meditation.wav</output_path>
  <content_type>music</content_type>
  <style>ambient</style>
  <duration>30</duration>
</tool_call>
```

### Local-Preferred Generation
```xml
<tool_call name="audio_generation" id="1">
  <text>High-energy electronic dance music</text>
  <output_path>music/edm_track.wav</output_path>
  <content_type>music</content_type>
  <style>electronic</style>
  <prefer_local>true</prefer_local>
</tool_call>
```

### Agent Workflow with Automatic Fallback
```
User: "Create background music for my podcast intro"

Agent: "I'll generate background music for your podcast intro."

<tool_call name="audio_generation" id="1">
  <text>Upbeat background music for technology podcast intro</text>
  <output_path>audio/podcast_bgm.wav</output_path>
  <content_type>music</content_type>
  <style>upbeat</style>
</tool_call>

[If Lyria 2 fails, agent automatically handles fallback]

Agent: "Lyria 2 is currently unavailable. I can set up local music generation for you. This will download a Docker container (~2GB) and take 2-3 minutes. Would you like me to proceed?"

User: "Yes, please set it up"

Agent: "Setting up local MusicGen... This may take a few minutes."

[CommandExecutionTool automatically sets up Docker container]

Agent: "Local music generation is now ready! Generating your background music..."

<tool_call name="audio_generation" id="2">
  <text>Upbeat background music for technology podcast intro</text>
  <output_path>audio/podcast_bgm.wav</output_path>
  <content_type>music</content_type>
  <style>upbeat</style>
</tool_call>

Agent: "✅ Background music generated successfully using local MusicGen! The file is saved as audio/podcast_bgm.wav"
```

### Long Content Generation
```xml
<tool_call name="audio_generation" id="3">
  <text><![CDATA[
This is a longer piece of content that might span multiple paragraphs.

It includes various punctuation marks, numbers like 123, and special characters.
The AudioGenerator will handle chunking this content appropriately for the provider's limits.
]]></text>
  <output_path>content/long_narration.wav</output_path>
  <voice>puck</voice>
</tool_call>
```

## Agent Conversation Flow

### Example Interaction

**User:** "Can you create an audio version of the README file for accessibility?"

**Agent Response:**
```
I'll read the README file and generate an audio version for you.

<tool_call name="read_file" id="1">
  <file_path>README.md</file_path>
</tool_call>

<tool_call name="audio_generation" id="2">
  <text>[README content will be inserted here after reading]</text>
  <output_path>audio/readme_narration.wav</output_path>
  <voice>zephyr</voice>
</tool_call>
```

**Tool Responses:**
```xml
<tool_response id="1">
  <tool_call name="read_file">
    <file_path>README.md</file_path>
  </tool_call>
  <result>
    <success>true</success>
    <data>
      <content># Project Title
This is the project description...</content>
    </data>
  </result>
</tool_response>

<tool_response id="2">
  <tool_call name="audio_generation">
    <text># Project Title
This is the project description...</text>
    <output_path>audio/readme_narration.wav</output_path>
    <voice>zephyr</voice>
  </tool_call>
  <result>
    <success>true</success>
    <message>Audio generated successfully</message>
    <data>
      <output_file_path>audio/readme_narration.wav</output_file_path>
      <file_size>156789</file_size>
      <duration_estimate>12.4s</duration_estimate>
    </data>
  </result>
</tool_response>
```

**Final Agent Message:**
"I've successfully created an audio version of your README file! The narration has been saved to `audio/readme_narration.wav` (156KB, approximately 12.4 seconds). You can now use this for accessibility purposes or to listen to your project documentation."

## Integration Points

### ToolManager Registration
The `AudioGenerationTool` will be registered in `ToolManager.initializeTools()` alongside existing tools.

### Environment Configuration
- Supports `GEMINI_API_KEY` environment variable
- Falls back to parameter-provided API key
- Future: Integration with `contaigents configure` command

### File System Integration
- Uses existing `FileService` patterns for path resolution
- Integrates with project file tree context
- Respects project boundaries for security

## Future Enhancements

### Additional Providers
- OpenAI TTS integration
- Azure Speech Services
- AWS Polly support

### Advanced Features
- Audio format options (MP3, WAV, OGG)
- Speed/pitch control parameters
- SSML support for advanced speech control
- Batch processing for multiple files

### Shared Memory Integration
- Store large audio files in shared memory system
- Reference audio files by memory IDs
- Enable audio processing workflows

## Testing Strategy

### Unit Tests
- Parameter validation
- AudioGenerator integration
- File path resolution
- Error handling scenarios

### Integration Tests
- End-to-end tool execution
- XML parsing and response formatting
- File system operations
- Provider API integration (with real keys)

### Agent Conversation Tests
- Multi-tool workflows (read file → generate audio)
- Error recovery scenarios
- Long content handling

## Benefits of Music Generation Integration

### 1. Complete Audio Production Pipeline
- **Speech + Music**: Create professional podcasts with background music
- **Layered Content**: Mix narration, music, and sound effects seamlessly
- **Consistent Quality**: All audio generated with compatible formats and sample rates

### 2. Creative Flexibility
- **Custom BGM**: Generate music tailored to specific content themes
- **Mood Matching**: Create music that matches the tone of speech content
- **Length Control**: Generate music of exact duration needed for projects

### 3. Cost Efficiency
- **No Licensing**: Generated music avoids copyright issues
- **Custom Creation**: No need to search for suitable existing music
- **Unlimited Usage**: Generate as much music as needed

### 4. Agent Intelligence
- **Context Awareness**: AI can generate appropriate music for content type
- **Style Matching**: Automatically select music styles that complement speech
- **Project Integration**: Seamlessly combine all audio elements

## Implementation Priority

### Phase 1: Foundation (Week 1-2)
- ✅ Extend AudioGenerationTool with content_type parameter
- ✅ Add music-specific parameters (style, duration)
- ✅ Update agent guidance with music examples

### Phase 2: Hybrid Music Generation (Week 3-4)
- 🔄 Implement Lyria 2 integration with existing Vertex AI client
- 🔄 Add local MusicGen fallback with Docker auto-setup
- 🔄 Integrate CommandExecutionTool for Docker management
- 🔄 Add smart provider selection logic
- 🔄 Implement user consent flow for local setup

### Phase 3: Enhanced Features (Week 5-6)
- 📋 Add sound effects generation (AudioGen)
- 📋 Implement style presets and templates
- 📋 Add music quality/length optimization
- 📋 Create music generation examples and documentation

### Phase 4: Production Ready (Week 7-8)
- 📋 Performance optimization
- 📋 Error handling and fallbacks
- 📋 Cloud deployment options
- 📋 Integration testing with AudioProjectTool

## Conclusion

## Benefits of Hybrid Approach

### 1. **Best of Both Worlds**
- **Cloud-First**: Always try high-quality Lyria 2 first
- **Local Fallback**: Automatic setup when cloud is unavailable
- **User Choice**: Option to prefer local generation
- **Seamless Experience**: Transparent fallback handling

### 2. **Reliability & Resilience**
- **No Single Point of Failure**: Multiple generation options
- **Offline Capability**: Local generation works without internet
- **API Quota Protection**: Fallback when quotas exceeded
- **Cost Control**: Local option for high-volume usage

### 3. **Intelligent Automation**
- **Auto-Detection**: Automatically detects when Docker setup needed
- **User Consent**: Asks permission before downloading large containers
- **Smart Caching**: Remembers user preferences
- **Error Recovery**: Graceful handling of setup failures

### 4. **Developer Experience**
- **Zero Configuration**: Works out of the box with existing setup
- **Progressive Enhancement**: Adds capabilities without breaking existing functionality
- **Tool Integration**: Leverages existing CommandExecutionTool
- **Transparent Operation**: Users don't need to understand the complexity

## Implementation Phases (Updated)

### Phase 1: Foundation (Week 1-2) ✅
- Extend AudioGenerationTool with content_type parameter
- Add music-specific parameters (style, negative_prompt, seed, prefer_local)
- Update agent guidance with hybrid examples

### Phase 2: Hybrid Implementation (Week 3-4) 🔄
- Implement Lyria 2 integration using existing Vertex AI client
- Add local MusicGen fallback with Docker auto-setup
- Integrate CommandExecutionTool for container management
- Add smart provider selection and user consent flows
- Test both cloud and local generation paths

### Phase 3: Enhanced Features (Week 5-6) 📋
- Add provider preference caching
- Implement health checks for local containers
- Add container management commands (start/stop/restart)
- Create comprehensive error handling and recovery
- Add performance monitoring and optimization

### Phase 4: Production Ready (Week 7-8) 📋
- Stress testing with both providers
- Documentation and user guides
- Integration testing with AudioProjectTool
- Performance benchmarking and optimization
- Production deployment and monitoring

This hybrid specification creates a robust, intelligent music generation system that provides users with the reliability of cloud services and the flexibility of local generation, all managed automatically through the existing contaigents tool architecture.