context-engineer-cli/README.md

A powerful CLI tool for assembling file contexts for LLM prompts with advanced pattern filtering, folder operations, and flexible output control

# Context Engineer CLI 🚀

A powerful CLI tool for assembling file contexts for LLM prompts and debugging sessions. Built with Bun/Node.js and TypeScript for speed and type safety.

## Features ✨

### Core Features
- **Interactive File Selection**: Enhanced fuzzy-search with folder operations
- **Pattern-Based Selection**: Use glob patterns for precise file matching
- **Folder-Level Operations**: Select/exclude entire folders efficiently
- **Smart Filtering**: Advanced include/exclude rules with pattern support
- **Token Counting**: Real-time GPT-like tokenization using OpenAI's tiktoken
- **Professional Output**: Clean XML-like format perfect for LLM contexts
- **Flexible Configuration**: JSON configs with full feature support
- **Optional Request Section**: Skip request prompts for code-only contexts
- **Performance Optimized**: Built for speed with Bun runtime

### Enhanced Features (v1.2+)
- **📁 Advanced File Selection**: Include/exclude patterns, folder-level controls
- **⚙️ Flexible Configuration**: JSON config files with pattern support
- **🚀 Multiple Operation Modes**: Interactive, pattern-based, or config-driven
- **🎯 Optional Request Section**: Use `--no-request` to skip prompts
- **🔍 Enhanced Interactive UI**: Folder navigation, batch operations, live search
- **🔄 Full Backward Compatibility**: All previous functionality preserved
- **🪟 Cross-Platform Path Support**: Works with both forward slashes (`/`) and backslashes (`\`) on Windows

## Installation 📦

### Prerequisites
- [Bun](https://bun.sh/) (recommended) or Node.js 16+

### Setup
```bash
# Clone or create the project
mkdir context-engineer-cli && cd context-engineer-cli

# Install dependencies
bun install
# or with npm: npm install
```

## Usage 🛠️

### Quick Start
```bash
# Interactive mode (original functionality)
context-engineer

# Pattern-based selection (new!)
context-engineer --include "src/**/*.ts" --exclude "**/*.test.ts"

# Skip request section (new!)
context-engineer --include "*.md" --no-request

# Use configuration file (enhanced!)
context-engineer --config context-config.json
```

### Enhanced Command Line Options

#### 📁 File Selection Options
- `-f, --files <files...>` - Specify files directly (original)
- `--include <patterns...>` - Include files matching glob patterns (new!)
- `--exclude <patterns...>` - Exclude files matching glob patterns (new!)
- `--include-folders <folders...>` - Include ALL files from specified folders (enhanced v1.2.2+)
- `--exclude-folders <folders...>` - Exclude specific folders (new!)

#### ⚙️ Configuration Options
- `-c, --config <file>` - Load configuration file (enhanced with JSON support)
- `-i, --interactive` - Force interactive mode with pre-filtering (new!)
- `--no-request, --skip-request` - Skip request section entirely (new!)
- `--include-project-structure` - Include project tree in output (default)
- `--no-project-structure, --skip-project-structure` - Skip project tree (new!)
- `-h, --help` - Show comprehensive help

### Usage Examples

#### Pattern-Based Selection (New!)
```bash
# Include TypeScript files, exclude tests
context-engineer --include "src/**/*.ts" --exclude "**/*.test.ts"

# Multiple include patterns
context-engineer --include "src/**/*.ts" --include "docs/**/*.md"

# Complex filtering
context-engineer --include "**/*.{ts,js,md}" --exclude "node_modules/**"
```

#### Folder-Based Operations (Enhanced!)
```bash
# Include ALL files from specific folders (v1.2.2+)
context-engineer --include-folders "src,docs,config"    # Includes every file in these folders

# Exclude folders
context-engineer --exclude-folders "tests,build,node_modules"

# Combined folder and pattern filtering (automatic deduplication)
context-engineer --include-folders "src" --exclude "**/*.test.ts"

# Cross-platform path support (v1.2.1+) - both work identically:
context-engineer --include-folders "web\\src\\utils"    # Windows-style backslashes
context-engineer --include-folders "web/src/utils"      # Unix-style forward slashes
```

**📁 Enhanced Folder Inclusion Behavior (v1.2.2)**:
- `--include-folders` now **automatically includes ALL files** within specified folders
- **Smart duplicate prevention**: Files matched by multiple methods are only included once
- **Works with patterns**: Combine folder inclusion with include/exclude patterns seamlessly
- **Interactive compatibility**: Folder operations work in both pattern and interactive modes

#### Skip Project Structure (New!)
```bash
# Generate context without project tree
context-engineer --include "src/**/*.ts" --no-project-structure

# Skip both request and project structure for minimal output
context-engineer --include "*.ts" --no-request --skip-project-structure
```

#### Enhanced Configuration Files (New!)
```bash
# Use JSON configuration (new format)
context-engineer --config context-config.json

# Override config with CLI arguments
context-engineer --config config.json --exclude "**/*.spec.ts"

# Legacy space-separated configs still work!
context-engineer --config legacy-config-file
```

#### Interactive Mode (Enhanced!)
```bash
# Default interactive mode
context-engineer

# Interactive with pre-filtering
context-engineer --include "src/**/*.ts" --interactive

# Interactive with folder pre-filtering
context-engineer --include-folders "src,docs" --interactive
```

### Enhanced Interactive Workflow

1. **File Discovery**: Scans directory with optional pre-filtering
2. **Enhanced File Selection**: New interactive features:
   - 📁 **Folder Operations**: Select entire folders at once
   - 🔍 **Smart Search**: Fuzzy search across files and folders
   - 👀 **Live Preview**: Real-time token counting and selection preview
   - 🗂️ **Organized Display**: Files grouped by folder structure
   - ⚡ **Batch Operations**: Clear all, folder selection, advanced filtering
3. **Optional Request**: Enter prompt text or skip with `--no-request`
4. **Output**: Get formatted `context-output.txt` file

### Configuration File Formats

#### New JSON Format (Recommended)
```json
{
  "include": ["src/**/*.ts", "docs/**/*.md"],
  "exclude": ["**/*.test.ts", "**/*.spec.ts"],
  "includeFolders": ["src", "docs"],    // Includes ALL files from these folders (v1.2.2+)
  "excludeFolders": ["tests", "node_modules"],
  "skipRequest": true,
  "includeProjectStructure": true
}
```

#### Legacy Format (Still Supported!) 
```
# Space-separated file list - still works perfectly!
src/brain.ts src/components/sense.ts docs/readme.md

# Or with newlines
src/brain.ts
src/components/sense.ts
docs/readme.md
```

**🔄 Backward Compatibility**: All existing config files and commands continue to work exactly as before!

### Output Formats

#### Standard Output (with request)
```xml
<project-structure>
[Project tree structure]
</project-structure>

<src/components/Button.tsx>
import React from 'react';
// ... file content
</src/components/Button.tsx>

<src/hooks/useTheme.ts>
// ... file content  
</src/hooks/useTheme.ts>

<request>
Please review this React component code and suggest improvements.
</request>
```

#### Code-Only Output (with --no-request)
```xml
<project-structure>
[Project tree structure]
</project-structure>

<src/components/Button.tsx>
import React from 'react';
// ... file content
</src/components/Button.tsx>

<src/hooks/useTheme.ts>
// ... file content  
</src/hooks/useTheme.ts>
```

#### Minimal Output (with --no-request --no-project-structure)
```xml
<src/components/Button.tsx>
import React from 'react';
// ... file content
</src/components/Button.tsx>

<src/hooks/useTheme.ts>
// ... file content  
</src/hooks/useTheme.ts>
```

## Operation Modes 🎯

### 1. Interactive Mode (Enhanced)
- **Default behavior**: `context-engineer`
- **Enhanced UI**: Folder operations, batch selection, live search
- **Pre-filtering**: Use patterns to narrow down initial file list
- **Best for**: Exploratory file selection, one-time contexts

### 2. Pattern-Based Mode (New!)
- **Direct selection**: Use include/exclude patterns
- **No interaction needed**: Perfect for automation
- **Example**: `context-engineer --include "src/**/*.ts" --exclude "**/*.test.ts"`
- **Best for**: Consistent file selection, CI/CD, scripted workflows

### 3. Configuration Mode (Enhanced)
- **JSON configs**: Full feature support with patterns and options
- **Legacy configs**: Space-separated file lists still supported
- **Example**: `context-engineer --config project-context.json`
- **Best for**: Project-specific contexts, team sharing, reproducible setups

### 4. Hybrid Mode (New!)
- **Pre-filter + Interactive**: Combine patterns with manual fine-tuning
- **Example**: `context-engineer --include "src/**/*.ts" --interactive`
- **Best for**: Starting with patterns, then customizing selection

## Migration Guide 🔄

### ✅ **All Previous Functionality Preserved**

**Your existing workflows continue to work exactly as before:**

```bash
# These commands work exactly the same as before
context-engineer                                    # ✅ Interactive mode
context-engineer -f file1.ts file2.ts             # ✅ Direct files
context-engineer --files src/main.ts               # ✅ Alternative syntax
context-engineer -c my-config-file                 # ✅ Config files
context-engineer --config legacy-space-separated   # ✅ Legacy configs
```

**Your existing config files work without any changes:**
- Space-separated file lists ✅
- Newline-separated file lists ✅
- Relative and absolute paths ✅
- All previous CLI flags ✅

### 🚀 **New Capabilities Available**

**You can now also use these enhanced features:**

```bash
# New pattern-based selection
context-engineer --include "src/**/*.ts" --exclude "**/*.test.ts"

# New folder operations
context-engineer --include-folders "src,docs" --exclude-folders "tests"

# Skip request section
context-engineer --no-request

# Skip project structure for minimal output
context-engineer --no-project-structure

# JSON configuration files
context-engineer --config enhanced-config.json
```

## Configuration ⚙️

You can modify the configuration at the top of `index.ts`:

```typescript
// Output filename
const OUTPUT_FILE = 'context-output.txt';

// Patterns to ignore during file discovery
const IGNORED_PATTERNS = [
  '**/node_modules/**',
  '**/.git/**',
  '**/dist/**',
  // ... add more patterns
];

// Tokenizer encoding (change for different models)
const TOKENIZER_ENCODING = 'cl100k_base';
```

## Development 👨‍💻

### Scripts
```bash
# Start the application
bun start

# Development mode with auto-reload
bun run dev
```

### Tech Stack
- **Runtime**: Bun (Node.js compatible)
- **Language**: TypeScript
- **CLI Framework**: Inquirer.js with autocomplete
- **File Operations**: Node.js fs/promises + glob
- **Tokenization**: js-tiktoken
- **Search**: Fuzzy matching

## Best Practices 💡

### For Large Projects
- **Use patterns** for consistent file selection: `--include "src/**/*.ts"`
- **Exclude test files** to focus on implementation: `--exclude "**/*.test.ts"`
- **Use folder operations** for quick bulk selection: `--include-folders "src"`
- **Monitor token counts** - most LLMs work best with <100k tokens
- **Use configuration files** for reproducible contexts

### For Better LLM Results
- **Be specific** in your request text or use `--no-request` for code-only contexts
- **Include relevant context** about what you're trying to achieve
- **Group related files** together (e.g., component + styles + types)
- **Use folder-based selection** for architectural overviews

### For Team Collaboration
- **Share JSON config files** for consistent project contexts
- **Use pattern-based selection** for reproducible results
- **Document common patterns** in your project README
- **Combine with CI/CD** for automated documentation generation

### Performance Tips
- **Use specific patterns** instead of broad wildcards when possible
- **Pre-filter with folders** before interactive selection
- **Save frequently used selections** as JSON configs
- **Use `--no-request`** when you only need file contents
- **Skip project structure** with `--no-project-structure` for minimal output

## Advanced Usage 🔧

### Combining Multiple Filters
```bash
# Complex filtering example
context-engineer \
  --include "src/**/*.{ts,tsx}" \
  --include "docs/**/*.md" \
  --exclude "**/*.test.ts" \
  --exclude "**/*.spec.tsx" \
  --exclude-folders "__tests__,coverage" \
  --no-request \
  --no-project-structure
```

### Configuration File Patterns
```json
{
  "include": [
    "src/**/*.{ts,tsx,js,jsx}",
    "docs/**/*.md",
    "*.{json,md}"
  ],
  "exclude": [
    "**/*.test.*",
    "**/*.spec.*",
    "**/*.d.ts",
    "**/dist/**",
    "**/build/**",
    "**/coverage/**"
  ],
  "excludeFolders": [
    "node_modules",
    ".git",
    "tests",
    "__tests__"
  ],
  "skipRequest": false,
  "includeProjectStructure": true
}
```

### Interactive Mode Enhancements
- **📁 Folder Navigation**: Browse and select entire folders
- **🔍 Enhanced Search**: Fuzzy search across files and folders
- **⚡ Batch Operations**: Select/deselect multiple items at once
- **👀 Live Preview**: Real-time token counting and selection summary
- **🗂️ Organized View**: Files grouped by directory structure

## Troubleshooting 🔧

### Common Issues

**"No files found matching patterns"**
- Check your glob patterns for typos
- Verify the folders exist in your current directory
- Try using `--interactive` to see what files are available
- Use simpler patterns first, then make them more specific

**"No files found" (Interactive mode)**
- Make sure you're in a directory with files
- Check if files are being filtered by ignore patterns
- Review the `IGNORED_PATTERNS` in the code

**"Binary file errors"**
- The tool automatically filters binary files
- Check the `BINARY_EXTENSIONS` list if you need to include specific types
- Binary files are skipped for safety and token efficiency

**"High token count warnings"**
- Use `--exclude` patterns to remove unnecessary files
- Focus on specific folders with `--include-folders`
- Consider using `--no-request` to save tokens
- Break large requests into smaller, focused contexts

**"Configuration file not working"**
- Verify JSON syntax if using the new format
- Check file paths are relative to current directory
- Legacy space-separated configs should still work
- Use `--help` to see configuration examples

### Performance Tips
- **For huge repositories**: Initial scan might take time, but it's a one-time cost
- **Fuzzy search optimization**: Search is fast even with thousands of files
- **Use specific patterns**: Reduces initial file discovery time
- **Folder pre-filtering**: Start with `--include-folders` for better performance

## License 📄

MIT License - feel free to use and modify!

## Contributing 🤝

Contributions welcome! Some ideas for improvements:
- Custom output formats (JSON, YAML, etc.)
- Integration with popular LLM APIs
- VS Code extension
- Directory-specific ignore files (.contextignore)
- File content preview in interactive mode
- Custom tokenizer support
- Git integration (changed files, branch diff contexts)
- Template system for common project types

See [ENHANCED_USAGE.md](./ENHANCED_USAGE.md) for detailed usage examples and advanced patterns.

---

Built with ❤️ for efficient context engineering. Happy prompting! 🎯