contaigents/AUDIO_PROJECT_TOOL_SPEC.md

Modular AI Content Ecosystem with Audio Generation

# Audio Project Tool Specification (Minimal MVP)

## Overview

This specification defines a minimal audio project tool that builds on the existing `AudioGenerationTool` to support basic multi-track audio mixing. The focus is on simple audio layering and export functionality using the current audio generation infrastructure.

## Current Audio Export System

The existing system works as follows:
1. **Gemini TTS API** returns audio data (usually raw PCM with MIME type `audio/l16`)
2. **AudioGenerator** automatically detects format and adds WAV headers if needed
3. **File Extension** is determined by MIME type (defaults to `.wav`)
4. **No MP3 conversion** - currently only supports WAV output format

### Supported Output Formats
- **WAV** (primary) - Raw PCM data with proper WAV headers
- **Auto-detection** based on provider MIME type
- **No MP3/other formats** - would require FFmpeg integration

## Minimal Tool Architecture

### Single Core Tool: AudioProjectTool
**Name**: `audio_project`
**Description**: Create simple multi-track audio projects and export to WAV
**Category**: Audio Project Management

## Minimal Tool Specification

### AudioProjectTool Parameters
| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `operation` | string | Yes | - | Operation: create, add_clip, mix_export |
| `project_path` | string | Yes | - | Path to project JSON file |
| `clips` | array | No | - | Array of audio clips with timing |
| `output_path` | string | No | - | Output file path (for mix_export) |

### Simple Project Format (JSON)
Instead of complex XML, use simple JSON structure:

```json
{
  "title": "My Audio Project",
  "clips": [
    {
      "file_path": "audio/intro.wav",
      "start_time": 0.0,
      "volume": 1.0
    },
    {
      "file_path": "audio/main.wav",
      "start_time": 30.0,
      "volume": 0.8
    }
  ]
}
```

### Tool Call Examples

**Create Project:**
```xml
<tool_call name="audio_project" id="1">
  <operation>create</operation>
  <project_path>projects/simple_podcast.json</project_path>
  <clips>
    [
      {
        "file_path": "audio/intro.wav",
        "start_time": 0.0,
        "volume": 1.0
      }
    ]
  </clips>
</tool_call>
```

**Add Clip:**
```xml
<tool_call name="audio_project" id="2">
  <operation>add_clip</operation>
  <project_path>projects/simple_podcast.json</project_path>
  <clips>
    [
      {
        "file_path": "audio/main_content.wav",
        "start_time": 25.0,
        "volume": 0.9
      }
    ]
  </clips>
</tool_call>
```

**Mix and Export (WAV only):**
```xml
<tool_call name="audio_project" id="3">
  <operation>mix_export</operation>
  <project_path>projects/simple_podcast.json</project_path>
  <output_path>output/final_podcast.wav</output_path>
</tool_call>
```

## Simple Audio Mixing Implementation

### How It Works
1. **Load Project JSON** - Read clip definitions and timing
2. **Load Audio Files** - Use existing AudioGenerator file reading
3. **Simple Buffer Mixing** - Layer audio buffers at specified times
4. **WAV Export** - Use existing WAV header creation from AudioGenerator

### Basic Mixing Algorithm
```typescript
// Pseudo-code for simple mixing
async function mixAudioProject(projectPath: string, outputPath: string) {
  const project = JSON.parse(await fs.readFile(projectPath));

  // Find total duration
  const totalDuration = Math.max(...project.clips.map(c =>
    c.start_time + getAudioDuration(c.file_path)
  ));

  // Create output buffer (assume 44.1kHz, 16-bit, mono)
  const sampleRate = 44100;
  const outputBuffer = new Float32Array(totalDuration * sampleRate);

  // Mix each clip
  for (const clip of project.clips) {
    const audioData = await loadWavFile(clip.file_path);
    const startSample = clip.start_time * sampleRate;

    // Simple additive mixing with volume
    for (let i = 0; i < audioData.length; i++) {
      outputBuffer[startSample + i] += audioData[i] * clip.volume;
    }
  }

  // Convert to 16-bit and save with WAV header
  const wavData = convertToWav(outputBuffer, sampleRate);
  await fs.writeFile(outputPath, wavData);
}
```

## Integration with Existing System

### Workflow Integration
1. **AudioGenerationTool** → Generate individual WAV files
2. **AudioProjectTool** → Create simple JSON project with clip timing
3. **AudioProjectTool** (mix_export) → Mix clips and export final WAV

### Simple Agent Workflow Example
```
User: "Create a podcast with intro and main content"

Agent: "I'll create a simple podcast project for you."

<tool_call name="audio_generation" id="1">
  <text>Welcome to our podcast about AI technology</text>
  <output_path>audio/intro.wav</output_path>
  <voice>zephyr</voice>
</tool_call>

<tool_call name="audio_generation" id="2">
  <text>Today we'll discuss the latest developments in machine learning and their impact on everyday life</text>
  <output_path>audio/main_content.wav</output_path>
  <voice>zephyr</voice>
</tool_call>

<tool_call name="audio_project" id="3">
  <operation>create</operation>
  <project_path>projects/my_podcast.json</project_path>
  <clips>
    [
      {
        "file_path": "audio/intro.wav",
        "start_time": 0.0,
        "volume": 1.0
      },
      {
        "file_path": "audio/main_content.wav",
        "start_time": 15.0,
        "volume": 0.9
      }
    ]
  </clips>
</tool_call>

<tool_call name="audio_project" id="4">
  <operation>mix_export</operation>
  <project_path>projects/my_podcast.json</project_path>
  <output_path>output/final_podcast.wav</output_path>
</tool_call>
```

## Limitations & Future Extensions

### Current Limitations
- **WAV only** - No MP3/other format support (would need FFmpeg)
- **Simple mixing** - Basic additive mixing, no advanced effects
- **No crossfades** - Clips play as-is without transitions
- **Mono output** - No stereo panning or multi-channel support

### Future Extensions (if needed)
- **FFmpeg integration** for MP3/other formats
- **Simple crossfade** between clips
- **Volume normalization** across clips
- **Basic fade in/out** for clips

## Implementation Plan

### Phase 1: Basic AudioProjectTool
1. **Create JSON project files** with clip timing information
2. **Add clips to existing projects** by updating JSON
3. **Simple WAV mixing** using existing AudioGenerator WAV handling
4. **Basic validation** of file paths and timing

### Phase 2: Enhanced Features (Optional)
1. **Volume normalization** across clips
2. **Simple fade in/out** for clips
3. **Basic crossfade** between overlapping clips
4. **FFmpeg integration** for MP3 export (if needed)

This minimal specification provides a practical starting point for audio project management while building on the existing, proven AudioGenerator infrastructure. The focus is on simplicity and immediate utility rather than comprehensive professional features.