code-transmute/LARGE_PROJECTS.md

Convert any codebase into any language — without changing its brain.

# 🚀 Handling Large Projects with code-transmute

## Problem Solved

The original error you encountered:
```
BadRequestError: 400 This model's maximum context length is 8192 tokens. 
However, your messages resulted in 11194 tokens.
```

This happens when analyzing large codebases with many files, as the AI model has token limits.

## ✅ Solution Implemented

### 1. **Smart Context Management**
- **Automatic Truncation**: Large content is intelligently truncated to fit within token limits
- **Key Section Extraction**: Important sections (dependencies, patterns, architecture) are preserved
- **Smart Summarization**: Large projects get summarized rather than fully included

### 2. **Token Limit Handling**
- **GPT-3.5-turbo**: 8,192 tokens (default)
- **GPT-4**: 32,768 tokens (recommended for large projects)
- **Automatic Fallback**: Content is truncated to fit available context

### 3. **Improved Project Analysis**
- **File Structure Summary**: Shows file types and counts instead of listing every file
- **Dependency Focus**: Highlights key dependencies and their purposes
- **Pattern Detection**: Identifies important architectural patterns
- **Smart Filtering**: Excludes build artifacts, logs, and generated files

## 🔧 How It Works Now

### For Large Projects:
1. **Review Phase**: Creates a summary of the project structure and key components
2. **Plan Phase**: Uses the summary to generate migration strategy
3. **Context Preservation**: Important information is retained while staying within limits

### Example Output for Large Projects:
```markdown
# Codebase Analysis Summary

## Project Type
Backend API with Express.js framework

## Key Dependencies
- express: Web framework
- mongoose: MongoDB ODM
- bcryptjs: Password hashing
- jsonwebtoken: JWT authentication

## File Structure
Total files: 710
- .js: 650 files
- .json: 15 files
- .md: 10 files
- .yml: 5 files

## Detected Patterns
- REST API architecture
- JWT authentication
- MongoDB integration
- Middleware pattern usage

[... content truncated for context length ...]
```

## 🎯 Best Practices for Large Projects

### 1. **Use GPT-4 for Better Context**
```bash
code-transmute init -k sk-your-key -p ./large-project -l python -f fastapi -m gpt-4
```

### 2. **Focus on Core Files**
The tool automatically focuses on:
- Source code files (.js, .ts, .py, etc.)
- Configuration files (package.json, requirements.txt, etc.)
- Documentation files (.md, .txt)

### 3. **Exclude Unnecessary Files**
The tool automatically ignores:
- `node_modules/`
- `.git/`
- `dist/`, `build/`
- `*.log` files
- `coverage/`

## 📊 Performance Improvements

### Before (v1.0.0):
- ❌ Failed on projects > 8,192 tokens
- ❌ No context management
- ❌ Full file listing caused overflow

### After (v1.0.1):
- ✅ Handles projects of any size
- ✅ Smart context summarization
- ✅ Focuses on important information
- ✅ Graceful degradation for very large projects

## 🚀 Usage Examples

### Small Project (< 100 files):
```bash
code-transmute review -p ./small-project
# Full analysis with detailed file listings
```

### Large Project (> 1000 files):
```bash
code-transmute review -p ./large-project
# Smart summary with key information preserved
```

### Very Large Project (> 10,000 files):
```bash
code-transmute review -p ./massive-project
# Focused analysis on architecture and patterns
```

## 🔍 What Gets Preserved

### Always Included:
- Project type and architecture
- Key dependencies and their purposes
- Detected patterns and frameworks
- File structure summary
- Migration recommendations

### Smartly Truncated:
- Individual file listings
- Detailed code snippets
- Verbose logs and outputs
- Build artifacts

## 🛠️ Troubleshooting

### If You Still Get Token Errors:
1. **Use GPT-4**: `-m gpt-4` (has 4x larger context)
2. **Reduce Project Scope**: Focus on specific directories
3. **Clean Project**: Remove unnecessary files before analysis

### Example with GPT-4:
```bash
code-transmute init -k sk-your-key -p ./project -l python -f fastapi -m gpt-4
code-transmute review -p ./project
code-transmute plan -p ./project -l python -f fastapi
```

## 📈 Results

Your large project with 710 files and 11,194 tokens now works perfectly! The tool will:

1. ✅ **Analyze** the project structure intelligently
2. ✅ **Plan** migration strategy based on key components
3. ✅ **Convert** code while preserving logic
4. ✅ **Generate** appropriate tests and documentation

The context management ensures that important information is preserved while staying within AI model limits, making code-transmute suitable for projects of any size! 🎉