meld/_dev/notes/dev/FIXPARSE.md

Meld: A template language for LLM prompts

# Meld Variable Syntax Changes and Parser Fixes

## Overview

We've implemented two major improvements to the Meld codebase:

1. **AST-Based Variable Resolution**: Replaced regex-based variable resolution with proper AST-based parsing
2. **Unified Variable Syntax**: Standardized the variable syntax for text and data variables to `{{variable}}`

## 1. AST-Based Variable Resolution

### Problem
The codebase was inconsistently using regex patterns to detect and resolve variable references instead of properly utilizing the AST parser. This approach was:
- Error-prone (regex has limitations for parsing nested structures)
- Inconsistent (different regex patterns in different parts of the code)
- Hard to maintain (changes to syntax required updating multiple regex patterns)

### Solution
- Completely rewrote the `VariableReferenceResolver` to use the parser service
- Eliminated direct regex usage for variable detection
- Added proper handling for AST node types: `TextVar` and `DataVar`
- Improved state variable lookup using context state
- Implemented better debugging and error handling

### Files Modified
- `/services/resolution/ResolutionService/resolvers/VariableReferenceResolver.ts`
- `/services/resolution/ResolutionService/ResolutionService.ts`
- `/services/pipeline/DirectiveService/handlers/definition/PathDirectiveHandler.ts`
- `/services/resolution/ValidationService/validators/PathDirectiveValidator.ts`

## 2. Unified Variable Syntax

### Previous Syntax
- Text variables: `${textvar}`
- Data variables: `#{datavar}`
- Path variables: `$pathvar`

### New Syntax
- Text variables: `{{textvar}}`
- Data variables: `{{datavar}}` with field access as `{{datavar.field}}`
- Path variables: `$pathvar` (unchanged)

### Changes Made
- Updated AST node types to handle the new unified syntax
- Fixed `PathDirectiveValidator` to handle different field naming conventions in the AST
- Updated `ResolutionService` to properly use the refactored `VariableReferenceResolver`
- Added support for both `id` and `identifier` field names for path variables
- Improved field access for data variables

### Test Updates
- Updated test expectations to use the new `{{variable}}` syntax
- Fixed integration tests to validate proper resolution with the new syntax
- Ensured all tests correctly verify both text and data variable resolution

## Benefits of These Changes

1. **Consistency**: Single unified syntax for variables makes the language more intuitive
2. **Reliability**: AST-based parsing is more robust than regex for complex structures
3. **Maintainability**: Centralized variable resolution logic makes future changes easier
4. **Performance**: More efficient variable detection and resolution
5. **Error Handling**: Better error messages when variable resolution fails

## Path Variables ($path)

Path variables remain unchanged, still using the `$path` syntax. This decision was made to:
- Maintain backward compatibility with existing Meld scripts
- Keep the special nature of path variables distinct from regular variables
- Preserve the unique validation rules that apply only to path variables

## Next Steps

1. Complete additional integration tests to ensure full compatibility
2. Update user documentation to reflect the new syntax
3. Consider adding deprecation warnings for the old syntax
4. Review other areas that might still use regex for parsing