@avdl/folder-walker/README.md

A powerful, fluent, and configurable directory traversal utility for Node.js with TypeScript support

# @avdl/folder-walker

A powerful, fluent, and configurable directory traversal utility for Node.js with TypeScript support.

## Features

- 🚀 **Fluent API** - Chain methods for intuitive configuration
- 📁 **Flexible traversal** - Files, folders, or both
- 🔍 **Smart filtering** - By extension, pattern, size, and more
- 📖 **Content reading** - Optional file content loading
- ⚡ **High performance** - Async iterators and parallel processing
- 🛡️ **Type-safe** - Full TypeScript support with great inference
- 0️⃣ **Zero dependencies** - Lightweight and secure

## Installation

```bash
npm install @avdl/folder-walker
# or
yarn add @avdl/folder-walker
```

## Quick Start

```typescript
import { FolderWalker } from '@avdl/folder-walker';

// Basic usage - iterate over all files
const walker = new FolderWalker();
for await (const file of walker.files('.')) {
  console.log(file.path);
}

// Advanced configuration with async operations
const configuredWalker = new FolderWalker()
  .withDepth(3)
  .withExtensions(['.ts', '.js'])
  .withFileContents('utf-8')
  .exclude(['node_modules', '.git'])
  .onFile(async ({ path, content }) => {
    // Async operations like saving to database, uploading, etc.
    await processFile(path, content);
    console.log(`Processed ${path}`);
  });

await configuredWalker.execute('./src');
```

## API Reference

### Configuration Methods

- `withDepth(n)` - Set maximum traversal depth
- `withFilesOnly()` - Only traverse files
- `withFoldersOnly()` - Only traverse folders  
- `withFilesAndFolders()` - Traverse both (default)
- `withFileContents(encoding)` - Read file contents
- `withExtensions(extensions)` - Filter by file extensions
- `withPattern(regex)` - Filter by filename pattern
- `exclude(patterns)` - Exclude paths
- `withHidden()` - Include hidden files
- `withSymlinks()` - Follow symbolic links
- `withMaxFileSize(bytes)` - Skip large files
- `withFilter(asyncFilter)` - Add custom async filter function

### Callbacks (Async Support)

- `onFile(handler)` - Process each file (supports async functions)
- `onFolder(handler)` - Process each folder (supports async functions)
- `onError(handler)` - Handle errors

### Result Collection

- `withFileCollector()` - Collect file handler results into array
- `withFileCollector(collectorFn)` - Transform collected results with custom function
- `withFolderCollector()` - Collect folder handler results into array
- `withFolderCollector(collectorFn)` - Transform collected results with custom function

### Execution

- `execute(path)` - Run with callbacks, returns void (streaming) or collected results
- `walker.files(path)` - Async iterator for files
- `walker.folders(path)` - Async iterator for folders
- `walker.entries(path)` - Async iterator for all entries

## Running Examples

Try the included examples to see folder-walker in action:

```bash
# List all available examples
npm run examples:list

# Run individual examples
npm run example:basic      # Basic usage patterns
npm run example:async      # Async operations  
npm run example:filters    # Custom filtering
npm run example:collectors # Type-safe collection

# Run all examples
npm run examples
```

## Examples

### Find all TypeScript files

```typescript
const walker = new FolderWalker()
  .withExtensions(['.ts', '.tsx'])
  .exclude(['node_modules', 'dist']);

for await (const file of walker.files('.')) {
  console.log(file.path);
}
```

### Process markdown files with async operations

```typescript
await new FolderWalker()
  .withExtensions(['.md'])
  .withFileContents('utf-8')
  .onFile(async ({ path, content }) => {
    const wordCount = content.split(/\s+/).length;
    await saveToDatabase(path, wordCount); // Async operation
    console.log(`${path}: ${wordCount} words`);
  })
  .execute('./docs');
```

### Type-safe result collection

```typescript
// Streaming mode (no collection) - returns void
await new FolderWalker()
  .onFile(({ path }) => {
    console.log(`Processing: ${path}`); // void return
  })
  .execute('./src');

// Collect results - returns ProcessResult[]
const results = await new FolderWalker()
  .onFile(({ path, stats }) => ({ path, size: stats.size })) // ProcessResult return
  .withFileCollector() // Enable collection
  .execute('./src');

// Custom collector transformation - returns ProjectSummary
const summary = await new FolderWalker()
  .onFile(({ path, stats }) => ({ path, size: stats.size }))
  .withFileCollector((files) => ({
    totalFiles: files.length,
    totalSize: files.reduce((sum, f) => sum + f.size, 0)
  })) // Transform to summary
  .execute('./src');
```

### Custom async filters

```typescript
await new FolderWalker()
  .withFileContents('utf-8')
  .withFilter(async (entry) => {
    // Custom filter: files containing specific content
    if (entry.isFile && entry.content) {
      const hasKeyword = (entry.content as string).includes('TODO');
      await logAnalysis(entry.path, hasKeyword); // Async operation
      return hasKeyword;
    }
    return false;
  })
  .withFilter((entry) => {
    // Multiple filters: also check file size
    return entry.stats.size < 10000;
  })
  .onFile(({ path }) => console.log(`TODO found in: ${path}`))
  .execute('./src');
```

### Concurrent file processing

```typescript
const tasks: Promise<void>[] = [];

await new FolderWalker()
  .onFile(async ({ path, stats }) => {
    if (stats.size > 1024 * 1024) { // 1MB
      // Process large files concurrently
      const task = processLargeFile(path);
      tasks.push(task);
    }
  })
  .execute('.');

// Wait for all concurrent operations
await Promise.all(tasks);
```

## Filter Order and Performance

Filters are applied in the following order for optimal performance:

1. **Built-in filters** (fastest, applied first):
   - Hidden files (`withHidden()`)
   - Include/exclude patterns
   - File extensions (`withExtensions()`)
   - Filename patterns (`withPattern()`)
   - File size limits (`withMaxFileSize()`)

2. **Custom filters** (`withFilter()`):
   - Applied in the order they were added
   - Support both sync and async operations
   - Can access full file metadata and content

**Performance Tips:**
- Place expensive filters (file content analysis) last
- Use built-in filters when possible (they're optimized)
- Custom filters receive fully-populated entry objects

## License

MIT