miniml/plans/load-models-from-directory.md

A minimal, embeddable semantic data modeling language for generating SQL queries from YAML model definitions. Inspired by LookML.

# Plan: Add loadModelsFromDirectory Function

## Overview
Add a new function `loadModelsFromDirectory` to scan a specified directory and all subdirectories for YAML model files, returning a dictionary of loaded MiniML models.

## Function Signature
```typescript
export async function loadModelsFromDirectory(directory: string): Promise<Record<string, MinimlModel>>
```

## Implementation Details

### 1. Directory Scanning
- Use Node.js `fs.readdir` with `{ recursive: true, withFileTypes: true }` to scan directories
- Filter for files with `.yaml` extension
- Handle both files and subdirectories recursively

### 2. Path Processing
- Convert absolute file paths to relative keys
- Remove base directory prefix and `.yaml` extension
- Normalize path separators to forward slashes for consistency
- Example: `/path/to/models/sales/revenue.yaml` → `"sales/revenue"`

### 3. Model Loading
- Use existing `loadModel` function for each discovered YAML file
- Leverage all existing model processing (validation, expansion, etc.)
- Maintain error context for individual file failures

### 4. Error Handling
- Validate directory exists and is accessible
- Provide meaningful errors for file access issues
- Include file path context in model loading errors
- Handle edge cases like empty directories gracefully

### 5. Return Format
- `Record<string, MinimlModel>` dictionary
- Keys: relative paths without `.yaml` extension
- Values: fully processed MiniML models

## Implementation Steps

1. **Examine current load.ts file and loadModel function implementation** ✅
2. **Design loadModelsFromDirectory function signature and implementation** ✅
3. **Implement directory scanning with recursive subdirectory support**
4. **Add path processing to create keys without .yaml extension**
5. **Handle error cases and edge conditions**
6. **Write comprehensive unit tests for new function**
7. **Update README.md with documentation for new function**

## Test Coverage Plan

### Basic Functionality
- Directory with single YAML file
- Directory with multiple YAML files
- Recursive subdirectory scanning
- Mixed file types (ignoring non-YAML files)

### Path Processing
- Verify key generation removes directory prefix
- Verify key generation removes `.yaml` extension
- Test path normalization (forward slashes)
- Test nested subdirectory paths

### Error Handling
- Invalid/non-existent directory
- Directory access permissions
- Individual model loading failures
- Empty directories

### Edge Cases
- Symbolic links to YAML files
- Files with multiple extensions (e.g., `.yaml.bak`)
- Very deep directory structures

## Usage Examples

### Basic Usage
```typescript
import { loadModelsFromDirectory } from 'miniml';

// Load all models from directory
const models = await loadModelsFromDirectory('./models');

// Access specific model
const salesModel = models['sales'];
const userModel = models['analytics/users'];
```

### Error Handling
```typescript
try {
    const models = await loadModelsFromDirectory('./models');
    console.log(`Loaded ${Object.keys(models).length} models`);
} catch (error) {
    console.error('Failed to load models:', error.message);
}
```

## Documentation Updates

### API Reference Section
Add new function documentation alongside existing `loadModel` and `loadModelSync`:

```markdown
#### `loadModelsFromDirectory(directory: string): Promise<Record<string, MinimlModel>>`
Recursively scans a directory for YAML model files and returns a dictionary of loaded models.

**Parameters:**
- `directory`: Path to the directory to scan

**Returns:**
- Promise resolving to a Record where keys are relative file paths (without .yaml extension) and values are loaded MinimlModel objects

**Example:**
```typescript
const models = await loadModelsFromDirectory('./models');
const salesModel = models['sales'];           // from ./models/sales.yaml
const userModel = models['analytics/users'];  // from ./models/analytics/users.yaml
```

### Usage Examples Section
Add example showing directory-based model loading for multi-model scenarios.

## Technical Considerations

### Dependencies
- Uses Node.js built-in `fs` and `path` modules
- No additional dependencies required
- Maintains compatibility with existing codebase patterns

### Performance
- Asynchronous directory scanning
- Parallel model loading where possible
- Efficient path processing using built-in Node.js utilities

### Consistency
- Follows existing code style (4-space indentation, TypeScript types)
- Uses same error handling patterns as existing functions
- Integrates with existing model processing pipeline

## Files to Modify

1. **lib/load.ts** - Add new function implementation
2. **test/load.test.ts** - Add comprehensive unit tests (create if doesn't exist)
3. **README.md** - Update API Reference and add usage examples
4. **lib/index.ts** - Export new function (if needed)

## Success Criteria

- Function correctly scans directories recursively
- Path keys are generated properly (no extension, relative paths)
- All existing model processing is preserved
- Comprehensive test coverage (>90%)
- Documentation is clear and includes examples
- No breaking changes to existing functionality