meld/_dev/notes/dev/FIXPATHSPARSE.md

Meld: A template language for LLM prompts

# Meld Path and Parsing Improvements

## Overview

This document consolidates key learnings and improvements made to the Meld codebase regarding:

1. **Path Variable Handling**: Changes in `meld-ast` that affected path representation and resolution
2. **Variable Syntax Unification**: Standardization of variable syntax and resolution mechanisms
3. **AST Structure Alignment**: Reconciliation of AST structure with validator expectations

These changes have significantly improved the robustness and maintainability of the codebase, addressing issues that were causing test failures and inconsistent behavior.

## 1. Path Resolution Improvements

### Key Issues Addressed

#### StructuredPath Format Transition
- **Before**: Paths were simple strings
- **Now**: Paths are objects with a structured format, including raw, normalized, and structured components

```typescript
interface StructuredPath {
  raw: string;
  normalized?: string;
  structured: {
    base: string;
    segments: string[];
    variables?: {
      text?: string[];
      path?: string[];
      special?: string[];
    };
  };
}
```

#### Property Name Mismatches
Path directive properties in the AST (`id`, `path`) differed from what validators expected (`identifier`, `value`):

| Directive | AST Property | Expected by Validator |
|-----------|-------------|------------------------|
| Path      | `id`        | `identifier`           |
| Define    | `name`      | `identifier`           |

#### Special Path Variables
Special path variables like `$PROJECTPATH`, `$HOMEPATH`, `$.`, and `$~` now receive proper validation and resolution.

### Implemented Fixes

1. **Updated ResolutionService.ts**:
   - Enhanced `resolveInContext` to handle both strings and StructuredPath objects
   - Added type declarations to prevent TypeScript errors
   - Improved handling of special path variables

2. **Updated PathDirectiveHandler.ts**:
   - Corrected handling of the StructuredPath object format
   - Improved special path variable handling (PROJECTPATH, HOMEPATH)
   - No longer mirroring path variables as text variables

3. **PathDirectiveValidator.ts Improvements**:
   - Updated error messages to match expected test formats
   - Enhanced validation for absolute paths and relative segments
   - Added appropriate error severity levels

## 2. Variable Syntax Unification

### Syntax Evolution

#### Previous Syntax
- Text variables: `${textvar}`
- Data variables: `#{datavar}`
- Path variables: `$pathvar`

#### New Unified Syntax
- Text variables: `{{textvar}}`
- Data variables: `{{datavar}}` with field access as `{{datavar.field}}`
- Path variables: `$pathvar` (unchanged)

### Key Improvements

1. **AST-Based Variable Resolution**:
   - Replaced regex-based resolution with proper AST parsing
   - Eliminated direct regex patterns for variable detection
   - Added handlers for AST node types: `TextVar` and `DataVar`

2. **VariableReferenceResolver Rewrite**:
   - Complete rewrite to use parser service instead of regex
   - Better state variable lookup using context
   - Improved error handling and debugging

3. **Path Variable Distinction**:
   - Path variables remain with `$pathvar` syntax
   - No longer mirrored as text variables
   - Kept distinct from text and data variables for clarity

## 3. Important Learnings

### 1. Service Consistency
Maintaining consistency between services that handle similar tasks is critical. When a feature like variable interpolation is implemented in multiple places, changes need to be applied uniformly.

### 2. Type-Aware String Conversion
Different data types require different string conversion strategies:
- Arrays: Comma-separated values (`text,data,path`)
- Objects: JSON serialization 
- Primitives: Simple string conversion

### 3. AST-Driven Development
The AST should drive the validation and handling layers, not vice versa. When the AST changes:
- Validators need to be updated to match the new structure
- Handlers need to align with the updated validators
- Integration tests may need adjustments

### 4. Variable Resolution Contexts
Different variable types have different resolution rules:
- Path variables: Only valid in path contexts (paths and commands)
- Text variables: Can be used in any text context
- Data variables: Can be used in text with field access

## 4. Implementation Strategy for AST Alignment

### Direct Codebase Alignment
Rather than creating adapter layers, we chose to directly align the codebase with the AST structure:

1. **For Path Directives**:
   - Update validators to check for `id` instead of `identifier`
   - Update handlers to work with `path` instead of `value`
   - Ensure proper handling of the structured path object

2. **For Variable Resolution**:
   - Utilize the AST parser for detecting variables
   - Standardize on the `{{var}}` syntax for text and data variables
   - Keep path variables as `$pathvar` for backward compatibility

3. **For Interface Consistency**:
   - Update interface definitions to match the AST structure
   - Ensure proper type checking throughout the resolution pipeline
   - Add robust error handling for type mismatches

## 5. Remaining Considerations

### Performance Implications
The new StructuredPath format and AST-based variable resolution may have performance implications that should be monitored.

### Backward Compatibility
While the codebase has been updated to handle the new formats:
- Legacy syntax for text variables (`${var}`) may still be found in existing scripts
- Data variable field access may use different formats

### Documentation Updates
User documentation should be updated to reflect:
- The new unified variable syntax
- The distinctions between path, text, and data variables
- Rules for variable usage in different contexts

## 6. Testing Strategy

Integration tests were critical in identifying and validating fixes:
- Path variable handling tests
- Text and data variable interpolation tests
- Field access tests for data variables
- Path validation tests

## 7. Existing Testing/Debugging Infrastructure

The testing and debugging infrastructure played a critical role in identifying, diagnosing, and resolving the path and parsing issues. 

### Integration Tests

The API module integration tests (`npm test api`) were instrumental in exposing mismatches between the AST structure and the validation/handling layers:

1. **Directive Validation Tests**: Revealed property name mismatches (e.g., `id` vs `identifier`)
2. **Variable Interpolation Tests**: Identified inconsistencies in variable resolution
3. **Path Resolution Tests**: Highlighted issues with the new structured path format
4. **Output Conversion Tests**: Showed discrepancies in how variables were processed

These tests provided clear expectations about how variable interpolation and path handling should work, serving as a guide for the implementation.

### Test Context Framework

The test context framework (`TestContext` class) provides comprehensive testing capabilities:

```typescript
async startDebugSession(config?: Partial<DebugSessionConfig>): Promise<string> {
  const defaultConfig: DebugSessionConfig = {
    captureConfig: {
      capturePoints: ['pre-transform', 'post-transform', 'error'] as const,
      includeFields: ['nodes', 'transformedNodes', 'variables'] as const,
      format: 'full'
    },
    visualization: {
      format: 'mermaid',
      includeMetadata: true,
      includeTimestamps: true
    },
    traceOperations: true,
    collectMetrics: true
  };
  // ...
}
```

Key features include:

1. **State Visualization**: The ability to visualize the state in Mermaid or DOT formats
2. **Debug Sessions**: Capturing pre-transform, post-transform, and error states
3. **Metrics Collection**: Performance and operation metrics for analysis
4. **In-Memory Filesystem**: Testing file operations without touching the real filesystem
5. **State Tracking**: Monitoring changes to the state during execution

### Mock Services

The test infrastructure includes mocked versions of core services:

1. **MockStateService**: For testing state operations in isolation
2. **MockResolutionService**: For testing resolution without dependencies
3. **MemfsTestFileSystem**: For simulating filesystem operations
4. **TestSnapshot**: For comparing filesystem states before and after operations

### Debugging Capabilities

Advanced debugging tools helped diagnose complex issues:

1. **State Diffing**: Comparing expected vs. actual state
2. **AST Inspection**: Examining the AST structure at various points
3. **Error Context**: Enhanced error reporting with context information
4. **Tracing**: Operation-by-operation tracing through the execution pipeline

### Test-Driven Development Approach

The tests served as both documentation and validation:

1. **Clear Expectations**: Tests defined expected behavior for variable handling
2. **Regression Prevention**: Ensured fixes didn't break existing functionality
3. **Edge Case Coverage**: Tests for special cases (arrays, nested objects, etc.)
4. **API Consistency**: Validated consistent behavior across different services

This robust testing and debugging infrastructure made it possible to systematically identify, diagnose, and fix the complex interplay of issues between the AST structure, validators, and handlers.

## Conclusion

These improvements have significantly enhanced the robustness of Meld's path handling and variable resolution systems. By aligning the codebase with the AST structure and standardizing on a unified variable syntax, we've reduced complexity and improved maintainability while ensuring backward compatibility where needed.