contaigents/VIDEO_SHOT_DETECTION.md

Modular AI Content Ecosystem with Audio Generation

# Video Shot Detection with FFmpeg Tool

## Overview

The FFmpeg tool now supports **video shot detection** - the ability to automatically identify scene changes and shot boundaries in videos. This is perfect for video analysis, content segmentation, and automated editing workflows.

## How It Works

Shot detection uses FFmpeg's built-in `scene` filter to analyze visual continuity between frames. When the visual difference between consecutive frames exceeds a threshold, it's marked as a shot boundary.

## Basic Usage

### 1. Detect Shot Boundaries

```bash
# Using the tool directly
{
  "operation": "detect_shots",
  "input_file": "video.mp4",
  "output_file": "shots.txt",
  "scene_threshold": "0.3"
}
```

**Output**: A text file with timestamps where shot boundaries occur:
```
2.518333
8.14
15.33
```

### 2. Split Video by Shots

The recommended workflow is a **two-step process**:

1. **First**: Detect shot boundaries
2. **Then**: Use trim operations to split the video

```bash
# Step 1: Detect shots
{
  "operation": "detect_shots",
  "input_file": "video.mp4", 
  "output_file": "shots.txt"
}

# Step 2: Split using detected timestamps
{
  "operation": "trim",
  "input_file": "video.mp4",
  "output_file": "segment_001.mp4",
  "start_time": "0",
  "end_time": "2.518333"
}

{
  "operation": "trim", 
  "input_file": "video.mp4",
  "output_file": "segment_002.mp4",
  "start_time": "2.518333",
  "end_time": "8.14"
}
```

## Parameters

### scene_threshold
- **Range**: 0.0 - 1.0
- **Default**: 0.3
- **Lower values** (0.1-0.2): More sensitive, detects subtle changes
- **Higher values** (0.4-0.6): Less sensitive, only major scene changes

### output_format
- **timestamps** (default): Simple list of shot boundary times
- **metadata**: Detailed information about each shot

## Practical Examples

### Content Analysis
```javascript
// Analyze a documentary for scene changes
{
  "operation": "detect_shots",
  "input_file": "documentary.mp4",
  "output_file": "scene_analysis.txt",
  "scene_threshold": "0.2"  // Sensitive to catch all transitions
}
```

### Highlight Reel Creation
```javascript
// Find major scene changes for highlights
{
  "operation": "detect_shots", 
  "input_file": "sports_game.mp4",
  "output_file": "highlights.txt",
  "scene_threshold": "0.5"  // Only major changes
}
```

### Automated Video Segmentation
```javascript
// Split a long video into manageable segments
{
  "operation": "detect_shots",
  "input_file": "webinar.mp4", 
  "output_file": "segments.txt",
  "scene_threshold": "0.3"
}
// Then use trim operations to create individual files
```

## Integration with Chat Service

The shot detection works seamlessly with the chat service:

```
"Analyze the video presentation.mp4 to find all the scene changes and then split it into separate files for each segment"
```

The agent will:
1. Run shot detection to find boundaries
2. Read the timestamps from the output file
3. Use trim operations to create individual segment files
4. Provide a summary of all created segments

## Use Cases

### 🎬 Video Editing Workflows
- **Auto-segmentation**: Automatically break long videos into scenes
- **Rough cut creation**: Identify natural break points for editing
- **Content organization**: Organize footage by detected scenes

### 📊 Content Analysis
- **Video structure analysis**: Understand pacing and scene distribution
- **Content summarization**: Extract key moments based on scene changes
- **Quality assessment**: Identify abrupt cuts or transitions

### 🤖 Automated Processing
- **Batch processing**: Process multiple videos with consistent segmentation
- **Thumbnail generation**: Create thumbnails at shot boundaries
- **Metadata extraction**: Generate scene-based metadata for videos

### 🎯 Specialized Applications
- **Sports analysis**: Detect play changes, camera switches
- **Educational content**: Split lectures by topic changes
- **Marketing videos**: Identify product showcases, testimonials

## Performance Characteristics

- **Speed**: ~400-500ms for typical videos (depends on length and complexity)
- **Accuracy**: High accuracy for clear scene changes, adjustable sensitivity
- **Resource usage**: Minimal - uses FFmpeg's optimized scene detection
- **File size**: Timestamp files are tiny (few KB), video segments vary by content

## Tips for Best Results

### Threshold Selection
- **0.1-0.2**: Use for subtle transitions, talking heads, interviews
- **0.3-0.4**: Good default for most content types
- **0.5-0.6**: Use for action videos, sports, high-motion content

### Workflow Optimization
1. **Test first**: Try different thresholds on a sample to find optimal settings
2. **Batch process**: Use the same threshold for similar content types
3. **Validate results**: Check a few segments to ensure quality
4. **Automate**: Build scripts that read timestamps and create segments

## Technical Details

The shot detection uses FFmpeg's `scene` filter with the following approach:
- Analyzes pixel differences between consecutive frames
- Applies configurable threshold to determine scene boundaries
- Outputs precise timestamps for programmatic use
- Maintains high performance through FFmpeg's optimized algorithms

This implementation provides a solid foundation for video analysis and automated editing workflows while maintaining simplicity and reliability.