meld/_dev/prompts/old/phase1-context.md

Meld: A template language for LLM prompts

<YourRole>
You are an expert in building reliable and maintainable DSL systems, particularly in structuring state interpreters.

You are passionate about SOLID architecture, taking methodical approaches, and making incremental and testable changes.

You're the architect for meld, an LLM prompt scripting language designed to make it easy to assemble complex context in modular ways using directives:

- @embed to embed text
- @run to run commands
- @define to create commands that can take parameters
- @path, @data, and @text variable types
- @import for importing variables and command definitions from other meld files

We have come a long way and you strongly want to ship meld. You're balancing pragmatism and a yagni spirit with a desire to have absolutely rock solid architecture that's going to serve as the foundation for a widely used open source project.
</YourRole>

# Meld Phase 1: Foundation Repair Context

We are working on fixing critical foundation issues in the Meld codebase, specifically focusing on path parsing/resolution and AST integration. We currently have 645 passing tests and 57 failing tests, many related to these foundation issues.

## Phase 1 Focus Areas

1. Path resolution and parsing issues (property name mismatches, structured path format, special variables)
2. AST integration and proper service architecture (making ParserService the sole interface to meld-ast)

\===============================
\=== SHIPPING PLAN FOR PHASE 1 =

# Meld Shipping Plan

## Overview

This document outlines our strategic plan for completing the Meld implementation and shipping a production-ready version. We currently have 645 passing tests and 57 failing tests, with several known issues documented in our planning files. Our primary measure of success is the API integration tests proving we can properly parse and build correct output from real-world Meld files.

## Strategic Approach

Our approach is organized into phases that build upon each other, with a focus on addressing fundamental issues first. Each phase includes a plan review component to ensure we adapt to newly discovered issues.

## Phase 1: Foundation Repair (3-4 days)

**Goal**: Fix the most critical infrastructure issues related to path parsing and AST handling that are causing test failures.

### 1.1 Path Resolution and Parsing (1-2 days)

**Tasks**:
- Fix the structured path format transition issues in ResolutionService
- Ensure proper handling of special path variables ($PROJECTPATH, $HOMEPATH, $., $~)
- Correct property name mismatches between AST nodes and validator expectations
- Update PathDirectiveHandler to properly handle the StructuredPath object format
- Update PathDirectiveValidator to align with expected test formats

**Success Criteria**:
- Path-related tests pass
- Path variables resolve correctly in different contexts
- Path directive validation works consistently

### 1.2 AST Integration and Service Architecture (1-2 days)

**Tasks**:
- Enforce ParserService as the sole interface to meld-ast
- Remove direct meld-ast imports from other services
- Remove custom code fence validation regex in favor of AST properties
- Update ContentResolver to leverage AST node properties
- Implement the most critical parts of Phase 1 from PLAN-REGEX.md

**Success Criteria**:
- ParserService correctly providing all needed AST functions
- Services properly using AST node properties
- No redundant regex for parsing where AST properties exist

### 1.3 Plan Review and Adjustment (0.5 days)

**Tasks**:
- Review passing and failing tests
- Document newly discovered issues
- Adjust priorities for Phase 2 based on findings
- Update the SHIP.md document with revised timelines

## Phase 2: Variable Resolution System (2-3 days)

**Goal**: Standardize the variable resolution system to use AST-based resolution consistently.

### 2.1 Resolution System Implementation (1-2 days)

**Tasks**:
- Refactor VariableReferenceResolver to use AST-based variable resolution
- Replace regex variable extraction with parser-based resolution
- Standardize variable syntax handling ({{var}} for text/data, $var for paths)
- Update CommandResolver to use the standardized resolution system

**Success Criteria**:
- Variable interpolation tests passing
- Consistent handling of all variable types
- Elimination of regex-based variable detection

### 2.2 Path and Variable Integration (1 day)

**Tasks**:
- Ensure path variables remain distinct from text variables
- Fix variable mirroring issues
- Implement proper context-aware variable resolution
- Update error messages related to variable resolution

**Success Criteria**:
- Integrated tests with both path and variable resolution passing
- Clear separation between variable types
- Proper error messages for variable-related issues

### 2.3 Plan Review and Adjustment (0.5 days)

**Tasks**:
- Review test outcomes
- Document newly discovered issues
- Adjust priorities for Phase 3
- Update SHIP.md with revised timelines

## Phase 3: Directive Validation and Handling (2-3 days)

**Goal**: Fix directive validators and handlers to work consistently with AST node properties.

### 3.1 Directive Validator Updates (1 day)

**Tasks**:
- Update ImportDirectiveValidator to handle structured path objects
- Fix DefineDirectiveValidator for property flexibility
- Update EmbedDirectiveValidator for consistency with AST
- Create shared validation utilities for identifiers

**Success Criteria**:
- Directive validation tests passing
- Consistent validation across all directive types
- Shared utilities reducing duplicate code

### 3.2 Directive Handler Implementation (1-2 days)

**Tasks**:
- Update ImportDirectiveHandler for path extraction
- Fix DefineDirectiveHandler for value resolution
- Update EmbedDirectiveHandler for path handling
- Complete updates to validate code fence blocks

**Success Criteria**:
- Directive handler tests passing
- Proper handling of all directive types
- Consistent handler implementation patterns

### 3.3 Plan Review and Adjustment (0.5 days)

**Tasks**:
- Review test outcomes
- Document newly discovered issues
- Adjust priorities for Phase 4
- Update SHIP.md with revised timelines

## Phase 4: API Completion and Integration (2-3 days)

**Goal**: Finalize the API and ensure all integration tests pass.

### 4.1 API Integration Test Fixes (1-2 days)

**Tasks**:
- Address remaining failing tests
- Fix Code Fence test fixtures
- Ensure proper output formatting
- Verify all directive types work end-to-end

**Success Criteria**:
- All API integration tests passing
- Consistent behavior across all test scenarios
- Proper error handling in all contexts

### 4.2 API Surface Refinement (1 day)

**Tasks**:
- Review and update API documentation
- Create or update API examples
- Ensure consistent naming and typing across API
- Implement any remaining performance optimizations

**Success Criteria**:
- Well-documented API with examples
- Consistent naming and typing
- Clear error types and handling documentation

### 4.3 Plan Review and Final Adjustment (0.5 days)

**Tasks**:
- Review overall test status
- Document any remaining issues
- Finalize priorities for Phase 5
- Update SHIP.md with revised timelines

## Phase 5: CLI Implementation (3-4 days)

**Goal**: Create a thin CLI wrapper on top of the completed API.

### 5.1 CLI Core Implementation (1-2 days)

**Tasks**:
- Create new CLI entry point
- Implement command-line argument parsing
- Map CLI options to API options
- Handle basic file I/O

**Success Criteria**:
- CLI successfully wrapping the API
- Proper handling of command-line arguments
- Correct mapping to API options

### 5.2 CLI-Specific Features (1 day)

**Tasks**:
- Implement watch mode
- Add version and help commands
- Handle stdout output
- Implement interactive prompts

**Success Criteria**:
- Watch mode working correctly
- Help and version commands giving correct output
- Proper handling of stdout

### 5.3 CLI Testing (1 day)

**Tasks**:
- Create CLI-specific tests
- Implement end-to-end tests
- Test error handling and exit codes
- Verify all CLI-specific features

**Success Criteria**:
- All CLI tests passing
- End-to-end tests verifying complete functionality
- Proper error handling in all scenarios

## Phase 6: Finalization and Release (1-2 days)

**Goal**: Prepare for release with documentation and migration planning.

### 6.1 Documentation Updates (0.5-1 day)

**Tasks**:
- Update all user-facing documentation
- Create or update tutorials
- Document the new unified variable syntax
- Document path handling rules and examples

**Success Criteria**:
- Complete and accurate documentation
- Clear tutorials for common use cases
- Documentation reflecting latest syntax and rules

### 6.2 Migration Strategy (0.5-1 day)

**Tasks**:
- Create migration guide for existing users
- Implement deprecation warnings
- Plan for backward compatibility
- Create release notes and timeline

**Success Criteria**:
- Clear migration path for existing users
- Documented breaking changes
- Comprehensive release notes

## Implementation Guidelines

Throughout all phases, we will adhere to these guidelines:

1. **Focus on Critical Path Issues First**
   - Prioritize fixes that unblock the most failing tests
   - Address foundational issues before surface-level ones
   - Target the most impactful services first

2. **Test-Driven Development**
   - Use failing tests to guide implementation
   - Add new tests for edge cases
   - Maintain high test coverage

3. **Maintain Type Safety**
   - Ensure proper TypeScript typing
   - Use interfaces for service interactions
   - Maintain strict type checking

4. **Leverage Existing Infrastructure**
   - Use our robust testing framework
   - Leverage debug services for complex issues
   - Use existing path handling capabilities

5. **Consistent Architecture**
   - Maintain clear service boundaries
   - Adhere to established patterns
   - Document architectural decisions

## Accounting for Emergent Issues

This plan explicitly acknowledges that we will uncover new issues as we progress. Our strategy for handling these:

1. **Phase Reviews**: Each phase includes a dedicated review step to assess progress and adjust priorities
2. **Living Document**: This SHIP.md will be updated after each phase review to reflect new findings
3. **Triage Process**: New issues will be categorized as:
   - **Critical**: Must be fixed in the current phase
   - **Important**: Should be addressed in the next phase
   - **Deferrable**: Can be addressed after initial release

## Total Timeline Estimate

- **Phase 1**: 3-4 days
- **Phase 2**: 2-3 days
- **Phase 3**: 2-3 days
- **Phase 4**: 2-3 days
- **Phase 5**: 3-4 days
- **Phase 6**: 1-2 days

**Total Estimate**: 13-19 days

This timeline includes the review steps and accounts for some discovery of new issues, but significant unexpected challenges could extend it further.

## Success Criteria

The implementation will be considered successful when:

1. All tests pass (currently aiming to fix 57 failing tests)
2. API integration tests prove proper parsing and output from real-world examples
3. Path handling works correctly in all contexts
4. Variable resolution is consistent with unified syntax
5. Service architecture is clean with proper boundaries
6. The API is well-documented with examples
7. The CLI successfully wraps the API with required functionality
8. A clear migration path exists for users

Regular updates to this document will track our progress toward these goals.

\===============================
\=== STRUCTURED PATH FORMAT ISSUES =

# Meld Path and Parsing Improvements

## Overview

This document consolidates key learnings and improvements made to the Meld codebase regarding:

1. **Path Variable Handling**: Changes in `meld-ast` that affected path representation and resolution
2. **Variable Syntax Unification**: Standardization of variable syntax and resolution mechanisms
3. **AST Structure Alignment**: Reconciliation of AST structure with validator expectations

These changes have significantly improved the robustness and maintainability of the codebase, addressing issues that were causing test failures and inconsistent behavior.

## 1. Path Resolution Improvements

### Key Issues Addressed

#### StructuredPath Format Transition
- **Before**: Paths were simple strings
- **Now**: Paths are objects with a structured format, including raw, normalized, and structured components

```typescript
interface StructuredPath {
  raw: string;
  normalized?: string;
  structured: {
    base: string;
    segments: string[];
    variables?: {
      text?: string[];
      path?: string[];
      special?: string[];
    };
  };
}
```

#### Property Name Mismatches
Path directive properties in the AST (`id`, `path`) differed from what validators expected (`identifier`, `value`):

| Directive | AST Property | Expected by Validator |
|-----------|-------------|------------------------|
| Path      | `id`        | `identifier`           |
| Define    | `name`      | `identifier`           |

#### Special Path Variables
Special path variables like `$PROJECTPATH`, `$HOMEPATH`, `$.`, and `$~` now receive proper validation and resolution.

### Implemented Fixes

1. **Updated ResolutionService.ts**:
   - Enhanced `resolveInContext` to handle both strings and StructuredPath objects
   - Added type declarations to prevent TypeScript errors
   - Improved handling of special path variables

2. **Updated PathDirectiveHandler.ts**:
   - Corrected handling of the StructuredPath object format
   - Improved special path variable handling (PROJECTPATH, HOMEPATH)
   - No longer mirroring path variables as text variables

3. **PathDirectiveValidator.ts Improvements**:
   - Updated error messages to match expected test formats
   - Enhanced validation for absolute paths and relative segments
   - Added appropriate error severity levels

## 2. Variable Syntax Unification

### Syntax Evolution

#### Previous Syntax
- Text variables: `${textvar}`
- Data variables: `#{datavar}`
- Path variables: `$pathvar`

#### New Unified Syntax
- Text variables: `{{textvar}}`
- Data variables: `{{datavar}}` with field access as `{{datavar.field}}`
- Path variables: `$pathvar` (unchanged)

### Key Improvements

1. **AST-Based Variable Resolution**:
   - Replaced regex-based resolution with proper AST parsing
   - Eliminated direct regex patterns for variable detection
   - Added handlers for AST node types: `TextVar` and `DataVar`

2. **VariableReferenceResolver Rewrite**:
   - Complete rewrite to use parser service instead of regex
   - Better state variable lookup using context
   - Improved error handling and debugging

3. **Path Variable Distinction**:
   - Path variables remain with `$pathvar` syntax
   - No longer mirrored as text variables
   - Kept distinct from text and data variables for clarity

## 3. Important Learnings

### 1. Service Consistency
Maintaining consistency between services that handle similar tasks is critical. When a feature like variable interpolation is implemented in multiple places, changes need to be applied uniformly.

### 2. Type-Aware String Conversion
Different data types require different string conversion strategies:
- Arrays: Comma-separated values (`text,data,path`)
- Objects: JSON serialization
- Primitives: Simple string conversion

### 3. AST-Driven Development
The AST should drive the validation and handling layers, not vice versa. When the AST changes:
- Validators need to be updated to match the new structure
- Handlers need to align with the updated validators
- Integration tests may need adjustments

### 4. Variable Resolution Contexts
Different variable types have different resolution rules:
- Path variables: Only valid in path contexts (paths and commands)
- Text variables: Can be used in any text context
- Data variables: Can be used in text with field access

## 4. Implementation Strategy for AST Alignment

### Direct Codebase Alignment
Rather than creating adapter layers, we chose to directly align the codebase with the AST structure:

1. **For Path Directives**:
   - Update validators to check for `id` instead of `identifier`
   - Update handlers to work with `path` instead of `value`
   - Ensure proper handling of the structured path object

2. **For Variable Resolution**:
   - Utilize the AST parser for detecting variables
   - Standardize on the `{{var}}` syntax for text and data variables
   - Keep path variables as `$pathvar` for backward compatibility

3. **For Interface Consistency**:
   - Update interface definitions to match the AST structure
   - Ensure proper type checking throughout the resolution pipeline
   - Add robust error handling for type mismatches

## 5. Remaining Considerations

### Performance Implications
The new StructuredPath format and AST-based variable resolution may have performance implications that should be monitored.

### Backward Compatibility
While the codebase has been updated to handle the new formats:
- Legacy syntax for text variables (`${var}`) may still be found in existing scripts
- Data variable field access may use different formats

### Documentation Updates
User documentation should be updated to reflect:
- The new unified variable syntax
- The distinctions between path, text, and data variables
- Rules for variable usage in different contexts

## 6. Testing Strategy

Integration tests were critical in identifying and validating fixes:
- Path variable handling tests
- Text and data variable interpolation tests
- Field access tests for data variables
- Path validation tests

## 7. Existing Testing/Debugging Infrastructure

The testing and debugging infrastructure played a critical role in identifying, diagnosing, and resolving the path and parsing issues.

### Integration Tests

The API module integration tests (`npm test api`) were instrumental in exposing mismatches between the AST structure and the validation/handling layers:

1. **Directive Validation Tests**: Revealed property name mismatches (e.g., `id` vs `identifier`)
2. **Variable Interpolation Tests**: Identified inconsistencies in variable resolution
3. **Path Resolution Tests**: Highlighted issues with the new structured path format
4. **Output Conversion Tests**: Showed discrepancies in how variables were processed

These tests provided clear expectations about how variable interpolation and path handling should work, serving as a guide for the implementation.

### Test Context Framework

The test context framework (`TestContext` class) provides comprehensive testing capabilities:

```typescript
async startDebugSession(config?: Partial<DebugSessionConfig>): Promise<string> {
  const defaultConfig: DebugSessionConfig = {
    captureConfig: {
      capturePoints: ['pre-transform', 'post-transform', 'error'] as const,
      includeFields: ['nodes', 'transformedNodes', 'variables'] as const,
      format: 'full'
    },
    visualization: {
      format: 'mermaid',
      includeMetadata: true,
      includeTimestamps: true
    },
    traceOperations: true,
    collectMetrics: true
  };
  // ...
}
```

Key features include:

1. **State Visualization**: The ability to visualize the state in Mermaid or DOT formats
2. **Debug Sessions**: Capturing pre-transform, post-transform, and error states
3. **Metrics Collection**: Performance and operation metrics for analysis
4. **In-Memory Filesystem**: Testing file operations without touching the real filesystem
5. **State Tracking**: Monitoring changes to the state during execution

### Mock Services

The test infrastructure includes mocked versions of core services:

1. **MockStateService**: For testing state operations in isolation
2. **MockResolutionService**: For testing resolution without dependencies
3. **MemfsTestFileSystem**: For simulating filesystem operations
4. **TestSnapshot**: For comparing filesystem states before and after operations

### Debugging Capabilities

Advanced debugging tools helped diagnose complex issues:

1. **State Diffing**: Comparing expected vs. actual state
2. **AST Inspection**: Examining the AST structure at various points
3. **Error Context**: Enhanced error reporting with context information
4. **Tracing**: Operation-by-operation tracing through the execution pipeline

### Test-Driven Development Approach

The tests served as both documentation and validation:

1. **Clear Expectations**: Tests defined expected behavior for variable handling
2. **Regression Prevention**: Ensured fixes didn't break existing functionality
3. **Edge Case Coverage**: Tests for special cases (arrays, nested objects, etc.)
4. **API Consistency**: Validated consistent behavior across different services

This robust testing and debugging infrastructure made it possible to systematically identify, diagnose, and fix the complex interplay of issues between the AST structure, validators, and handlers.

## Conclusion

These improvements have significantly enhanced the robustness of Meld's path handling and variable resolution systems. By aligning the codebase with the AST structure and standardizing on a unified variable syntax, we've reduced complexity and improved maintainability while ensuring backward compatibility where needed.

\===============================
\=== SERVICE ARCHITECTURE ISSUES =

# Meld Regex Replacement Plan

## Phase 1: Enforce Proper Service Architecture and Parser Integration

**Goal**: Establish ParserService as the sole interface to meld-ast and eliminate redundant regex

1. Refactor `ParserService` to fully utilize meld-ast features
   - Remove custom code fence validation regex in `validateCodeFences()` method
   - Use native AST properties for language and content information
   - Properly document and use CodeFenceNode.language and CodeFenceNode.content fields
   - Ensure ParserService provides all needed AST node functions to other services

2. Enforce architectural boundaries
   - Make ParserService the only component that directly imports 'meld-ast'
   - Remove direct 'meld-ast' imports from all other services
   - Ensure all other services receive MeldNode objects from ParserService
   - Create utility methods in ParserService for node type operations needed by other services

3. Update documentation on meld-ast capabilities and proper service architecture
   - Clarify how meld-ast already handles code fences, headings, and section detection
   - Document ParserService as the sole interface to 'meld-ast'
   - Update architecture documentation to reflect proper service boundaries

4. Train developers on proper AST usage patterns
   - Create examples demonstrating proper AST node inspection
   - Document best practices for accessing MeldNode properties via ParserService

**Timeline**: 1-2 weeks

## Phase 2: Resolution System Standardization

**Goal**: Replace manual variable detection with consistent resolution system

1. Refactor `VariableReferenceResolver`
   - Remove regex `/\{\{([^}]+)\}\}/g` variable extraction
   - Implement AST-based variable resolution exclusively
   - Deprecate `resolveSimpleVariables` method

2. Update `CommandResolver`
   - Replace regex `/\${([^}]+)}/g` with standard variable interpolation
   - Use resolution system for parameter replacement
   - Add structured command parameter handling

3. Standardize `ContentResolver`
   - Remove backtick extraction regex
   - Use AST node properties for code fence handling

4. Update `OutputService` to leverage AST node properties
   - Simplify and standardize `nodeToMarkdown` method
   - Replace direct content manipulation with AST node property access
   - Eliminate duplicate methods like `codeFenceToMarkdown` and `codeFenceToLLM`

**Timeline**: 2-3 weeks

## Phase 3: Directive Validation Standardization

**Goal**: Move string validation from regex to structured validators

1. Create shared validator for identifiers
   - Replace regex `/^[a-zA-Z0-9_]+$/` in PathDirectiveValidator
   - Replace regex `/^[a-zA-Z_][a-zA-Z0-9_]*$/` in TextDirectiveValidator
   - Create common validation utility class

2. Implement structured string tokenizer
   - Replace quote detection regex `/(?<!\\)['"`]/g`
   - Create proper string tokenizer for validation

3. Update directive format validators
   - Update ImportDirectiveValidator to use structured parsing
   - Remove complex regex patterns for bracket extraction
   - Create common bracketed content parser utility

**Timeline**: 2-3 weeks

## Phase 4: Directive Handler Refactoring

**Goal**: Make handlers use structured data from AST

1. Refactor `ImportDirectiveHandler`
   - Remove regex for path and import extraction
   - Use AST node structure directly
   - Share logic with ImportDirectiveValidator

2. Update `RunDirectiveHandler`
   - Remove regex for command extraction
   - Use structured directive data from nodes

3. Standardize format handling across handlers
   - Ensure all handlers use consistent approach
   - Create shared utilities for common operations

**Timeline**: 2-3 weeks

## Phase 5: Testing & Documentation

**Goal**: Ensure comprehensive test coverage and documentation updates

1. Create test suite for proper AST usage
   - Verify all node types are correctly accessed
   - Test code fence properties usage
   - Validate proper AST traversal patterns

2. Test resolution system changes
   - Ensure variable interpolation works correctly
   - Verify command parameter handling
   - Test complex nested scenarios

3. Create improved AST documentation
   - Create detailed documentation of meld-ast capabilities
   - Add examples showing proper node type access patterns
   - Document all available node properties for each node type
   - Create AST Explorer utility for visualizing node structures

4. Update existing documentation
   - Revise architecture documentation
   - Document best practices for handling different node types
   - Create developer guidelines for AST usage

**Timeline**: 2-3 weeks

## Implementation Strategy

1. **Service architecture first**: Enforce proper service boundaries and dependencies
2. **Incremental approach**: Update one service at a time, starting with the most fundamental (ParserService)
3. **Maintain compatibility**: Keep backward compatibility where possible during transition
4. **Test-driven development**: Write tests before implementing changes
5. **Consistent patterns**: Establish and document consistent patterns for all services

## Prioritization

1. `ParserService` - Most impactful as the sole interface to meld-ast
2. `VariableReferenceResolver` - Critical for variable resolution
3. `ResolutionService` - Affects multiple downstream services
4. `OutputService` - Important for ensuring proper node property usage
5. Validator classes - Important for consistent syntax validation
6. Directive handlers - Final implementation of the pattern

## Architectural Principles

1. **Single Responsibility**: Each service has one clear function
2. **Dependency Isolation**: Only ParserService should import and use meld-ast directly
3. **Interface Stability**: Services should communicate through well-defined interfaces
4. **Type Safety**: Leverage TypeScript types from meld-spec throughout the codebase
5. **Documentation**: Document node types and properties for developers

This phased approach ensures the codebase systematically moves from regex-based parsing to proper AST handling while maintaining proper service boundaries and architectural principles.

\===============================
\=== CORE ARCHITECTURE ==========

# Meld Architecture

## INTRODUCTION

Meld is a specialized, directive-based scripting language designed for embedding small "@directives" inside an otherwise plain text (e.g., Markdown-like) document. The code in this repository implements:

• Meld grammar rules and token types (e.g., text directives, path directives, data directives).
• The parsing layer that converts Meld content into an AST (Abstract Syntax Tree).
• A directive interpretation layer that processes these AST nodes and manipulates internal "states" to store variables and more.
• A resolution layer to handle variable references, path expansions, data manipulations, etc.
• Testing utilities and an in-memory FS (memfs) to simulate filesystems for thorough testing.

The main idea:
1. Meld code is parsed to an AST.
2. Each directive node is validated and interpreted, updating a shared "state" (variables, data structures, commands, etc.).
3. Optional transformations (e.g., output formatting) generate final representations (Markdown, LLM-friendly XML, etc.).

Below is an overview of the directory and service-level architecture, referencing code from this codebase.

## DIRECTORY & FILE STRUCTURE

At a high level, the project is arranged as follows (select key entries included):

project-root/
 ├─ api/                    ← High-level API and tests
 │   ├─ api.test.ts
 │   └─ index.ts
 ├─ bin/                    ← CLI entry point
 │   └─ meld.ts
 ├─ cli/                    ← CLI implementation
 │   ├─ cli.test.ts
 │   └─ index.ts
 ├─ core/                   ← Core utilities and types
 │   ├─ config/            ← Configuration (logging, etc.)
 │   ├─ errors/            ← Error class definitions
 │   │   ├─ MeldError.ts
 │   │   ├─ ServiceInitializationError.ts   ← Service initialization errors
 │   │   └─ ... other errors
 │   ├─ types/             ← Core type definitions
 │   │   ├─ dependencies.ts  ← Service dependency definitions
 │   │   └─ index.ts
 │   └─ utils/             ← Logging and utility modules
 │       ├─ logger.ts
 │       ├─ serviceValidation.ts  ← Service validation utilities
 │       └─ simpleLogger.ts
 ├─ services/              ← Core service implementations
 │   ├─ pipeline/          ← Main transformation pipeline
 │   │   ├─ ParserService/     ← Initial parsing
 │   │   ├─ InterpreterService/← Pipeline orchestration
 │   │   ├─ DirectiveService/  ← Directive handling
 │   │   │   ├─ handlers/
 │   │   │   │   ├─ definition/   ← Handlers for definition directives
 │   │   │   │   └─ execution/    ← Handlers for execution directives
 │   │   │   └─ errors/
 │   │   └─ OutputService/    ← Final output generation
 │   ├─ state/             ← State management
 │   │   ├─ StateService/      ← Core state management
 │   │   └─ StateEventService/ ← Core event system
 │   ├─ resolution/        ← Resolution and validation
 │   │   ├─ ResolutionService/ ← Variable/path resolution
 │   │   ├─ ValidationService/ ← Directive validation
 │   │   └─ CircularityService/← Circular dependency detection
 │   ├─ fs/                ← File system operations
 │   │   ├─ FileSystemService/ ← File operations
 │   │   ├─ PathService/      ← Path handling
 │   │   └─ PathOperationsService/ ← Path utilities
 │   └─ cli/               ← Command line interface
 │       └─ CLIService/    ← CLI entry point
 ├─ tests/                  ← Test infrastructure
 │   ├─ fixtures/          ← Test fixture data
 │   ├─ mocks/             ← Test mock implementations
 │   └─ utils/             ← Test utilities and helpers
 │       ├─ debug/         ← Test debug utilities
 │       │   ├─ StateDebuggerService/
 │       │   ├─ StateVisualizationService/
 │       │   ├─ StateHistoryService/
 │       │   └─ StateTrackingService/
 │       ├─ FixtureManager.ts
 │       ├─ MemfsTestFileSystem.ts
 │       ├─ ProjectBuilder.ts
 │       ├─ TestContext.ts
 │       └─ TestSnapshot.ts
 ├─ docs/                   ← Documentation
 ├─ package.json
 ├─ tsconfig.json
 ├─ tsup.config.ts
 └─ vitest.config.ts

Key subfolders:
• services/pipeline/: Core transformation pipeline services (parsing, interpretation, directives, output)
• services/state/: State management and event services
• services/resolution/: Resolution, validation, and circularity detection services
• services/fs/: File system, path handling, and operations services
• services/cli/: Command line interface services
• core/: Central types, errors, and utilities used throughout the codebase
• tests/utils/: Test infrastructure including debug utilities, memfs implementation, fixture management, and test helpers
• api/: High-level public API for using Meld programmatically
• cli/: Command line interface for Meld

## CORE LIBRARIES & THEIR ROLE

### meld-ast
   • parse(content: string): MeldNode[]
   • Basic parsing that identifies directives vs. text nodes.
   • Produces an AST which other services manipulate.

### llmxml
   • Converts content to an LLM-friendly XML format or can parse partially.
   • OutputService may call it if user requests "llm" format.

### meld-spec
   • Contains interface definitions for MeldNode, DirectiveNode, TextNode, etc.
   • Contains directive kind enumerations.

## HIGH-LEVEL FLOW

Below is a simplified flow of how Meld content is processed:

   ┌─────────────────────────────┐
   │   Meld Source Document      │
   └─────────────────────────────┘
                │
                ▼
   ┌─────────────────────────────┐
   │ ParserService.parse(...)    │
   │   → uses meld-ast to parse  │
   └─────────────────────────────┘
                │ AST (MeldNode[])
                ▼
   ┌─────────────────────────────────────────────────┐
   │ InterpreterService.interpret(nodes, options)    │
   │   → For each node, pass to DirectiveService     │
   │   → Handles node transformations                │
   └─────────────────────────────────────────────────┘
                │
                ▼
   ┌──────────────────────────────────────────┐
   │ DirectiveService                         │
   │   → Routes to correct directive handler  │
   │   → Handlers can provide replacements    │
   └──────────────────────────────────────────┘
                │
                ▼
   ┌───────────────────────────────────────────────┐
   │ StateService + ResolutionService + Others     │
   │   → Stores variables and transformed nodes    │
   │   → Path expansions, data lookups, etc.       │
   └───────────────────────────────────────────────┘
                │
                ▼
   ┌──────────────────────────────────────────┐
   │ OutputService                            │
   │   → Uses transformed nodes for output    │
   │   → Generates clean, directive-free      │
   │     markdown, LLM XML, or other formats  │
   └──────────────────────────────────────────┘

## MAJOR SERVICES (OVERVIEW)

Below are the key "services" in the codebase. Each follows the single responsibility principle:

### CLIService
   - Provides command-line interface for running Meld
   - Handles file watching and reprocessing
   - Manages format selection and output options
   - Routes to appropriate services based on CLI flags

### ParserService
   - Wraps the meld-ast parse(content) function
   - Adds location information with file paths (parseWithLocations)
   - Produces an array of MeldNode objects

### DirectiveService
   - Routes directives to the correct directive handler
   - Validates directives using ValidationService
   - Calls ResolutionService for variable resolution
   - Updates StateService with directive execution results
   - Supports node transformation through DirectiveResult interface
   - Handlers can provide replacement nodes for transformed output

### InterpreterService
   - Orchestrates the main interpret(nodes) pipeline
   - For each AST node:
       a) If it's text, store it or pass it along
       b) If it's a directive:
          - Calls DirectiveService for processing
          - Handles node transformations if provided
          - Updates state with transformed nodes
   - Maintains the top-level process flow
   - Supports transformation mode through feature flags

### StateService
   - Stores variables in maps:
       • textVars (for @text)
       • dataVars (for @data)
       • pathVars (for @path)
       • commands (for @define)
   - Tracks both original and transformed MeldNodes
   - Provides transformation capabilities for directive processing
   - Maintains transformation state during cloning
   - Provides child states for nested imports
   - Supports immutability toggles

### ResolutionService
   - Handles all variable interpolation:
       • Variables ("{{var}}", "{{data.field}}")
       • Path expansions ("$HOMEPATH/path")
       • Command references
   - Context-aware resolution
   - Circular reference detection
   - Sub-fragment parsing support

### CircularityService
   - Prevents infinite import loops
   - Detects circular variable references
   - Maintains dependency graphs

### PathService
   - Validates and normalizes paths
   - Enforces path security constraints
   - Handles path joining and manipulation
   - Supports test mode for path operations

### ValidationService
   - Validates directive syntax and constraints
   - Provides extensible validator registration
   - Throws MeldDirectiveError on validation failures
   - Tracks available directive kinds

###  FileSystemService
    - Abstracts file operations (read, write)
    - Supports both real and test filesystems
    - Handles path resolution and validation

### OutputService
    - Converts final AST and state to desired format
    - Uses transformed nodes when available
    - Supports markdown and LLM XML output
    - Integrates with llmxml for LLM-friendly formatting
    - Handles format-specific transformations
    - Provides clean output without directive definitions

## TESTING INFRASTRUCTURE

All tests are heavily reliant on a memory-based filesystem (memfs) for isolation and speed. The major testing utilities include:

### MemfsTestFileSystem
   – Thin wrapper around memfs
   – Offers readFile, writeFile, mkdir, etc. with in-memory data
   – Provides an ephemeral environment for all test IO

### TestContext
   – Central test harness that creates a new MemfsTestFileSystem
   – Provides references to all major services (ParserService, DirectiveService, etc.)
   – Allows writing files, snapshotting the FS, and comparing

### TestSnapshot
   – Takes "snapshots" of the current Memfs FS, storing a Map<filePath, content>
   – Compares snapshots to detect added/removed/modified files

### ProjectBuilder
   – Creates mock "projects" in the in-memory FS from JSON structure
   – Useful for complex, multi-file tests or large fixture-based testing

### Node Factories
   – Provides helper functions for creating AST nodes in tests
   – Supports creating directive, text, and code fence nodes
   – Includes location utilities for source mapping

Testing Organization:
• tests/utils/: Core test infrastructure (MemFS, snapshots, contexts)
• tests/mocks/: Minimal mocks and test doubles
• tests/fixtures/: JSON-based test data
• tests/services/: Service-specific integration tests

Testing Approach:
• Each test uses a fresh TestContext or recreates MemfsTestFileSystem
• Direct imports from core packages (meld-ast, meld-spec) for types
• Factory functions for creating test nodes and data
• Snapshots for tracking filesystem changes

## DEBUGGING INFRASTRUCTURE

The codebase includes specialized debugging services located in `tests/utils/debug/` that help diagnose and troubleshoot state-related issues:

### StateDebuggerService
   - Provides debug session management and diagnostics
   - Tracks state operations and transformations
   - Offers operation tracing and analysis
   - Helps identify state manipulation issues

### StateVisualizationService
   - Generates visual representations of state
   - Creates Mermaid/DOT graphs of state relationships
   - Visualizes state metrics and transformations
   - Aids in understanding complex state changes

### StateHistoryService
   - Records chronological state changes
   - Maintains operation history
   - Tracks transformation chains
   - Enables state change replay and analysis

### StateTrackingService
   - Monitors state relationships and dependencies
   - Tracks state lineage and inheritance
   - Records metadata about state changes
   - Helps debug scope and inheritance issues

Debugging Approach:
• Services can be enabled selectively in tests
• Debug output includes detailed state snapshots
• Visual representations help understand complex states
• History tracking enables step-by-step analysis

These debugging services are particularly useful for:
• Troubleshooting complex state transformations
• Understanding directive processing chains
• Analyzing variable resolution paths
• Debugging scope inheritance issues
• Visualizing state relationships

## SERVICE RELATIONSHIPS

Services in Meld follow a strict initialization order and dependency graph:

1. Base Services:
   - FileSystemService (no dependencies)
   - PathService (depends on FS)

2. State Management:
   - StateEventService (no dependencies)
   - StateService (depends on events)

3. Core Pipeline:
   - ParserService (independent)
   - ResolutionService (depends on State, FS)
   - ValidationService (depends on Resolution)
   - CircularityService (depends on Resolution)

4. Pipeline Orchestration:
   - DirectiveService (depends on multiple services)
   - InterpreterService (orchestrates others)

5. Output Generation:
   - OutputService (depends on State)

6. Debug Support:
   - DebuggerService (optional, depends on all)

Service initialization and validation is handled through the core/types/dependencies.ts system,
which ensures services are created in the correct order and all dependencies are satisfied.

## EXAMPLE USAGE SCENARIO

1) Input: A .meld file with lines like:
   @text greeting = "Hello"
   @data config = { "value": 123 }
   @import [ path = "other.meld" ]

2) We load the file from disk.
3) ParserService → parse the content → AST.
4) InterpreterService → interpret(AST).
   a) For each directive, DirectiveService → validation → resolution → update StateService.
   b) If an import is encountered, CircularityService ensures no infinite loops.
5) Once done, the final StateService has textVars.greeting = "Hello", dataVars.config = { value: 123 }, etc.
6) OutputService can generate the final text or an LLM-XML representation.

## ERROR HANDLING

• MeldDirectiveError thrown if a directive fails validation or interpretation.
• MeldParseError if the parser cannot parse content.
• PathValidationError for invalid paths.
• ResolutionError for variable resolution issues.
• MeldError as a base class for other specialized errors.

These errors typically bubble up to the caller or test.

## CONCLUSION

This codebase implements the entire Meld language pipeline:
• Parsing Meld documents into an AST.
• Validating & interpreting directives.
• Storing data in a hierarchical state.
• Resolving references (text, data, paths, commands).
• (Optionally) generating final formatted output.

Plus, it has a robust test environment with an in-memory FS, snapshots, and a test harness (TestContext) for integration and unit tests. Everything is layered to keep parsing, state management, directive logic, and resolution separate, adhering to SOLID design principles.

The ASCII diagrams, modules, and file references in this overview represent the CURRENT code as it is: multiple specialized services collaborating to parse and interpret Meld scripts thoroughly—test coverage is facilitated by the in-memory mocking and snapshot-based verification.

\===============================
\=== RELEVANT CODE =============

# ResolutionService.ts

```typescript
import * as path from 'path';
import { IStateService } from '@services/state/StateService/IStateService.js';
import { IResolutionService, ResolutionContext, ResolutionErrorCode } from './IResolutionService.js';
import { TextResolver } from './resolvers/TextResolver.js';
import { DataResolver } from './resolvers/DataResolver.js';
import { PathResolver } from './resolvers/PathResolver.js';
import { CommandResolver } from './resolvers/CommandResolver.js';
import { ContentResolver } from './resolvers/ContentResolver.js';
import { VariableReferenceResolver } from './resolvers/VariableReferenceResolver.js';
import { resolutionLogger as logger } from '@core/utils/logger.js';
import { IFileSystemService } from '@services/fs/FileSystemService/IFileSystemService.js';
import { IParserService } from '@services/pipeline/ParserService/IParserService.js';
import type { MeldNode, DirectiveNode, TextNode, DirectiveKind, CodeFenceNode, StructuredPath } from 'meld-spec';
import { MeldFileNotFoundError } from '@core/errors/MeldFileNotFoundError.js';
import { MeldResolutionError } from '@core/errors/MeldResolutionError.js';
import { ErrorSeverity } from '@core/errors/MeldError.js';
import { inject, singleton } from 'tsyringe';
import { CommandContextService } from '../../command/CommandContextService';
import { MeldInterpreterError } from '../../../errors';
import {
  ResolutionContext as DeprecatedContext,
  CommandParameterResolutionContext,
  NestedResolution
} from '../ResolutionContextFactory';
import { ICommandService } from '../../command/ICommandService';
import { Command, ParameterResolutionMap } from '../../command/Command';
import { CommandParameter } from '../../../types';

/**
 * Internal type for heading nodes in the ResolutionService
 * This is converted from TextNode when we detect a heading pattern
 */
interface InternalHeadingNode {
  content: string;
  level: number;
}

/**
 * Convert a TextNode to an InternalHeadingNode if it matches heading pattern
 * Returns null if the node is not a heading
 */
function parseHeadingNode(node: TextNode): InternalHeadingNode | null {
  const headingMatch = node.content.match(/^(#{1,6})\s+(.+)$/);
  if (!headingMatch) {
    return null;
  }
  return {
    level: headingMatch[1].length,
    content: headingMatch[2].trim()
  };
}

/**
 * Check if a node is a text node that represents a heading
 */
function isHeadingTextNode(node: MeldNode): node is TextNode {
  return node.type === 'Text' && (node as TextNode).content.match(/^#{1,6}\s+.+$/) !== null;
}

/**
 * Service responsible for resolving variables, commands, and paths in different contexts
 */
export class ResolutionService implements IResolutionService {
  private textResolver: TextResolver;
  private dataResolver: DataResolver;
  private pathResolver: PathResolver;
  private commandResolver: CommandResolver;
  private contentResolver: ContentResolver;
  private variableReferenceResolver: VariableReferenceResolver;

  constructor(
    private stateService: IStateService,
    private fileSystemService: IFileSystemService,
    private parserService: IParserService
  ) {
    this.textResolver = new TextResolver(stateService);
    this.dataResolver = new DataResolver(stateService);
    this.pathResolver = new PathResolver(stateService);
    this.commandResolver = new CommandResolver(stateService);
    this.contentResolver = new ContentResolver(stateService);
    // Create the variable reference resolver with the parser
    this.variableReferenceResolver = new VariableReferenceResolver(
      stateService,
      this,
      parserService
    );
  }

  /**
   * Parse a string into AST nodes for resolution
   */
  private async parseForResolution(value: string): Promise<MeldNode[]> {
    try {
      const nodes = await this.parserService.parse(value);
      return nodes || [];
    } catch (error) {
      // If parsing fails, treat the value as literal text
      return [{
        type: 'Text',
        content: value
      } as TextNode];
    }
  }

  /**
   * Resolve text variables in a string
   */
  async resolveText(text: string, context: ResolutionContext): Promise<string> {
    const nodes = await this.parseForResolution(text);
    return this.textResolver.resolve(nodes[0] as DirectiveNode, context);
  }

  /**
   * Resolve data variables and fields
   */
  async resolveData(ref: string, context: ResolutionContext): Promise<any> {
    const nodes = await this.parseForResolution(ref);
    return this.dataResolver.resolve(nodes[0] as DirectiveNode, context);
  }

  /**
   * Resolve path variables
   */
  async resolvePath(path: string, context: ResolutionContext): Promise<string> {
    logger.debug('Resolving path', { path, context });
    const nodes = await this.parseForResolution(path);
    return this.pathResolver.resolve(nodes[0] as DirectiveNode, context);
  }

  /**
   * Resolve command references
   */
  async resolveCommand(cmd: string, args: string[], context: ResolutionContext): Promise<string> {
    const node: DirectiveNode = {
      type: 'Directive',
      directive: {
        kind: 'run',
        name: cmd,
        identifier: cmd,
        args
      }
    };
    return this.commandResolver.resolve(node, context);
  }

  /**
   * Resolve content from a file path
   */
  async resolveFile(path: string): Promise<string> {
    if (!await this.fileSystemService.exists(path)) {
      throw new MeldFileNotFoundError(path);
    }
    return this.fileSystemService.readFile(path);
  }

  /**
   * Resolve raw content nodes, preserving formatting but skipping comments
   */
  async resolveContent(nodes: MeldNode[], context: ResolutionContext): Promise<string> {
    if (!Array.isArray(nodes)) {
      // If a string path is provided, read the file
      const path = String(nodes);
      if (!await this.fileSystemService.exists(path)) {
        throw new MeldResolutionError(
          `File not found: ${path}`,
          {
            code: ResolutionErrorCode.INVALID_PATH,
            details: { value: path },
            severity: ErrorSeverity.Fatal
          }
        );
      }
      return this.fileSystemService.readFile(path);
    }

    // Otherwise, process the nodes
    return this.contentResolver.resolve(nodes, context);
  }

  /**
   * Resolve any value based on the provided context rules
   */
  async resolveInContext(value: string | StructuredPath, context: ResolutionContext): Promise<string> {
    // Add debug logging for debugging path handling issues
    console.log('*** ResolutionService.resolveInContext', {
      value: typeof value === 'string' ? value : value.raw,
      allowedVariableTypes: context.allowedVariableTypes,
      pathValidation: context.pathValidation,
      stateExists: !!context.state,
      specialPathVar