@webdevtoday/grok-cli/docs/file-reference-system.md

A sophisticated CLI tool for interacting with xAI Grok 4, featuring conversation history, file reference, custom commands, memory system, and genetic development workflows

# File Reference System

## Overview

The File Reference System provides intelligent file discovery, search, and reference capabilities through `@` commands. It enables quick navigation and context-aware file operations.

## Features

- **Smart Search** - Fuzzy matching and relevance scoring
- **Git Integration** - Awareness of version-controlled files
- **Extension Filtering** - Search by file types
- **Recent Files** - Quick access to recently modified files
- **Preview Support** - Content preview for text files
- **Auto-completion** - Intelligent file suggestions

## Usage

### Basic File Search

```bash
# Search for files by name
@package.json
@cli.ts
@README

# Search with partial matches
@pack        # Matches package.json, package-lock.json, etc.
@src/cli     # Matches src/cli.ts, src/cli.js, etc.
```

### Special Commands

```bash
# Recently modified files (last 15)
@recent

# Git tracked files
@git

# Files by extension
@ext:ts      # All TypeScript files
@ext:py      # All Python files
@ext:md      # All Markdown files

# Get help and examples
@
```

### Path-based References

```bash
# Exact path reference
@src/core/config.ts
@docs/README.md
@.grok/settings.json

# Directory browsing
@src/
@docs/
```

## Search Algorithm

### Relevance Scoring

Files are ranked by relevance score based on:

1. **Exact name match** (100 points)
2. **Name starts with query** (50 points)
3. **Name contains query** (25 points)
4. **Path contains query** (10 points)
5. **Fuzzy match** (5 points)
6. **Git tracked** (+2 points)
7. **Recently modified** (+3 points if < 7 days, +2 if < 1 day)
8. **Important extensions** (+1 point for ts, js, py, md, json, yaml, yml)

### Fuzzy Matching

Supports fuzzy matching where characters can be non-consecutive:
- `pcj` matches `package.json`
- `clits` matches `cli.ts`
- `rdm` matches `README.md`

## File Information Display

### Single File Found

When one file matches, displays detailed information:

```
📁 File Details:
Path: src/cli.ts
Size: 15.2 KB
Modified: 2h ago
Type: file
Extension: .ts
Git tracked: Yes

📄 Content Preview:
  1: #!/usr/bin/env node
  2: 
  3: /**
  4:  * Grok CLI - Main entry point
  5:  * A sophisticated CLI tool for interacting with xAI Grok 4
  6:  */
  7: 
  8: import { Command } from 'commander';
  9: import chalk from 'chalk';
  10: import { ConfigManager } from '@/core/config';
... 5 more lines
```

### Multiple Files Found

When multiple files match, shows a list:

```
📁 Found 5 files matching "config":
  1. 📝 src/core/config.ts (3.2 KB)
  2. 📄 .grok/settings.json (1.1 KB)
  3. 📝 tsconfig.json (892 B)
  4. 📄 package.json (2.1 KB)
  5. 📝 docs/configuration.md (4.5 KB)

Use @<exact-path> to view a specific file
```

### Recent Files Display

```
📁 Recent Files:
  1. 📝 src/cli.ts (15.2 KB, 2h ago)
  2. 📄 package.json (2.1 KB, 3h ago)
  3. 📝 src/core/history.ts (8.9 KB, 4h ago)
  4. 📄 README.md (12.3 KB, 1d ago)
  5. 📝 src/types/index.ts (5.7 KB, 2d ago)
```

## Architecture

### FileReferenceManager

```typescript
class FileReferenceManager {
  private baseDir: string;
  private gitTrackedFiles: Set<string>;
  private fileCache: Map<string, FileReference>;
  private lastCacheUpdate: number;
  
  // Core search functionality
  async searchFiles(query: string, options?: FileSearchOptions): Promise<FileReference[]>
  
  // Autocomplete support
  async getFileSuggestions(partial: string): Promise<FileReference[]>
  
  // Specialized queries
  async getRecentFiles(limit?: number): Promise<FileReference[]>
  async getGitTrackedFiles(): Promise<FileReference[]>
  async getFilesByExtension(extension: string): Promise<FileReference[]>
  
  // File details
  async getFileDetails(path: string): Promise<FileReference | null>
}
```

### FileReference Interface

```typescript
interface FileReference {
  path: string;           // Absolute path
  relativePath: string;   // Relative to base directory
  name: string;          // File basename
  size: number;          // File size in bytes
  modified: Date;        // Last modification time
  type: 'file' | 'directory';
  extension?: string;    // File extension (without dot)
  isGitTracked?: boolean; // Whether file is tracked by git
}
```

### Search Options

```typescript
interface FileSearchOptions {
  pattern?: string;           // Search pattern
  maxResults?: number;        // Maximum results (default: 50)
  includeDirectories?: boolean; // Include directories (default: false)
  extensions?: string[];      // Filter by extensions
  excludePatterns?: string[]; // Exclude patterns
  searchDepth?: number;       // Maximum search depth
  sortBy?: 'name' | 'modified' | 'size' | 'relevance'; // Sort criteria
}
```

## Caching System

### File Cache

- **Cache Duration**: 30 seconds
- **Auto-refresh**: Updates automatically when needed
- **Git Integration**: Tracks git status changes
- **Performance**: Avoids filesystem hits for repeated searches

### Cache Invalidation

Cache is updated when:
- 30 seconds have passed since last update
- Search is performed after cache timeout
- Git status changes are detected

### Excluded Patterns

Default exclusions for performance:
```typescript
const excludePatterns = [
  'node_modules/**',
  '.git/**',
  'dist/**',
  'build/**',
  'coverage/**',
  '.next/**',
  '.nuxt/**',
  '.vscode/**',
  '.idea/**'
];
```

## Git Integration

### Git Awareness

- Automatically detects git repositories
- Tracks which files are version-controlled
- Provides git-specific views (`@git`)
- Uses git status for file metadata

### Git Commands Used

```bash
git ls-files  # List tracked files
```

### Fallback Behavior

When git is not available or not a git repository:
- Gracefully falls back to filesystem-only operation
- No git-specific features are available
- Search and other features continue working

## Performance Optimization

### Efficient File Discovery

1. **Globby Integration**: Uses fast glob patterns
2. **Streaming**: Processes files as they're discovered
3. **Filtering**: Early filtering to reduce processing
4. **Caching**: Aggressive caching with smart invalidation

### Search Performance

- **Indexed Search**: Pre-computed file metadata
- **Fuzzy Matching**: Optimized string matching algorithms
- **Result Limiting**: Configurable result limits
- **Lazy Loading**: Only loads details when needed

## Content Preview

### Supported File Types

Text files with preview support:
- `.txt`, `.md`, `.json`
- `.js`, `.ts`, `.py`
- `.html`, `.css`, `.yml`, `.yaml`
- `.xml`, `.csv`, `.log`
- `.sh`, `.bash`

### Preview Rules

- **Size Limit**: Files under 10KB only
- **Line Limit**: Shows first 10 lines
- **Encoding**: UTF-8 text files only
- **Error Handling**: Graceful fallback for binary files

## Integration with CLI

### Command Processing

File reference commands are processed in `handleFileReference()`:

1. **Parse Query**: Extract command and parameters
2. **Route Command**: Dispatch to appropriate handler
3. **Execute Search**: Perform file discovery
4. **Format Output**: Display results with formatting
5. **Handle Errors**: Provide helpful error messages

### Output Formatting

- **Colors**: Syntax highlighting with chalk
- **Icons**: File type indicators (📝 for git, 📄 for regular)
- **Sizes**: Human-readable file sizes
- **Times**: Relative timestamps (2h ago, 1d ago)

## Configuration

### Base Directory

```typescript
// Default: current working directory
const manager = new FileReferenceManager();

// Custom base directory
const manager = new FileReferenceManager('/path/to/project');

// Change at runtime
manager.setBaseDirectory('/new/path');
```

### Search Limits

```typescript
const options: FileSearchOptions = {
  maxResults: 20,        // Limit results
  searchDepth: 5,        // Limit directory depth
  includeDirectories: true, // Include folders
  extensions: ['ts', 'js'], // Filter extensions
};
```

## Error Handling

### Graceful Degradation

- **Permission Errors**: Skip inaccessible files
- **Git Errors**: Fall back to filesystem operations
- **Cache Errors**: Rebuild cache automatically
- **Search Errors**: Provide helpful error messages

### Error Types

```typescript
// File not found
No files found matching "nonexistent"

// Permission denied
Error searching files: Permission denied

// Git not available
Git tracked files not available (not a git repository)
```

## Advanced Usage

### Power User Features

```bash
# Search with wildcards (handled by fuzzy matching)
@src/*config*

# Multiple extension search
@ext:js @ext:ts    # Would need multiple commands

# Complex path matching
@src/core/config   # Finds src/core/config.ts
```

### Integration with Tools

The file reference system integrates with other Grok CLI tools:

- **Read Tool**: `@filename` → Read tool execution
- **Write Tool**: File path completion
- **Git Tool**: Status-aware file listing
- **Custom Commands**: File reference in workflows

## Future Enhancements

### Planned Features

- **Multi-extension search**: `@ext:ts,js,py`
- **Date filtering**: `@modified:today`, `@modified:week`
- **Size filtering**: `@size:>1MB`, `@size:<100KB`
- **Content search**: `@contains:"function main"`
- **Workspace awareness**: Multi-project support
- **Symbol search**: `@symbol:function`, `@symbol:class`

### Performance Improvements

- **Background indexing**: Continuous cache updates
- **Incremental search**: Real-time result updates
- **Search history**: Remember common searches
- **Smart caching**: Context-aware cache strategies