meld/_dev/prompts/old/arch.md

Meld: A template language for LLM prompts

You are an expert in building reliable and maintainable DSL systems, particularly in structuring state interpreters.

You are passionate about SOLID architecture, taking methodical approaches, and making incremental and testable changes.

We want to create a thoughtfully structured plan for addressing some complex issues we have encountered. We are now asking for your help preparing a detailed plan which is written in order to maximize success based on it being carried out by an LLM developer.

I am going to provide you with some context:

- Architecture documentation (slightly outdated)
- Test setup
- Issues we encountered
- Our completed plan to fix these issues
- The subsequent test failures we encountered as we reached the point of finishing that plan
- The advice you provided for strategically approaching resolving these issues

\======= CONTEXT

\=== ARCHITECTURE

# Meld Architecture

## INTRODUCTION

Meld is a specialized, directive-based scripting language designed for embedding small "@directives" inside an otherwise plain text (e.g., Markdown-like) document. The code in this repository implements:

• Meld grammar rules and token types (e.g., text directives, path directives, data directives).
• The parsing layer that converts Meld content into an AST (Abstract Syntax Tree).
• A directive interpretation layer that processes these AST nodes and manipulates internal "states" to store variables and more.
• A resolution layer to handle variable references, path expansions, data manipulations, etc.
• Testing utilities and an in-memory FS (memfs) to simulate filesystems for thorough testing.

The main idea:
1. Meld code is parsed to an AST.
2. Each directive node is validated and interpreted, updating a shared "state" (variables, data structures, commands, etc.).
3. Optional transformations (e.g., output formatting) generate final representations (Markdown, LLM-friendly XML, etc.).

Below is an overview of the directory and service-level architecture, referencing code from this codebase.

## DIRECTORY & FILE STRUCTURE

At a high level, the project is arranged as follows (select key entries included):

project-root/
 ├─ api/                    ← High-level API and tests
 │   ├─ api.test.ts
 │   └─ index.ts
 ├─ bin/                    ← CLI entry point
 │   └─ meld.ts
 ├─ cli/                    ← CLI implementation
 │   ├─ cli.test.ts
 │   └─ index.ts
 ├─ core/                   ← Core utilities and types
 │   ├─ config/            ← Configuration (logging, etc.)
 │   ├─ errors/            ← Error class definitions
 │   ├─ types/             ← Core type definitions
 │   └─ utils/             ← Logging and utility modules
 ├─ services/              ← Core service implementations
 │   ├─ CLIService/
 │   ├─ CircularityService/
 │   ├─ DirectiveService/
 │   │   ├─ handlers/
 │   │   │   ├─ definition/   ← Handlers for definition directives
 │   │   │   └─ execution/    ← Handlers for execution directives
 │   │   └─ errors/
 │   ├─ FileSystemService/
 │   ├─ InterpreterService/
 │   ├─ OutputService/
 │   ├─ ParserService/
 │   ├─ PathService/
 │   ├─ ResolutionService/
 │   │   ├─ resolvers/       ← Individual resolution handlers
 │   │   └─ errors/
 │   ├─ StateService/
 │   └─ ValidationService/
 │       └─ validators/      ← Individual directive validators
 ├─ tests/                  ← Test infrastructure
 │   ├─ fixtures/          ← Test fixture data
 │   ├─ mocks/             ← Test mock implementations
 │   ├─ services/          ← Service-specific tests
 │   └─ utils/             ← Test utilities and helpers
 │       ├─ FixtureManager.ts
 │       ├─ MemfsTestFileSystem.ts
 │       ├─ ProjectBuilder.ts
 │       ├─ TestContext.ts
 │       └─ TestSnapshot.ts
 ├─ docs/                   ← Documentation
 ├─ package.json
 ├─ tsconfig.json
 ├─ tsup.config.ts
 └─ vitest.config.ts

Key subfolders:
• services/: Each service is a self-contained module with its implementation, interface, tests, and any service-specific utilities
• core/: Central types, errors, and utilities used throughout the codebase
• tests/utils/: Test infrastructure including the memfs implementation, fixture management, and test helpers
• api/: High-level public API for using Meld programmatically
• cli/: Command line interface for Meld

## CORE LIBRARIES & THEIR ROLE

### meld-ast
   • parse(content: string): MeldNode[]
   • Basic parsing that identifies directives vs. text nodes.
   • Produces an AST which other services manipulate.

### llmxml
   • Converts content to an LLM-friendly XML format or can parse partially.
   • OutputService may call it if user requests "llm" format.

### meld-spec
   • Contains interface definitions for MeldNode, DirectiveNode, TextNode, etc.
   • Contains directive kind enumerations.

## HIGH-LEVEL FLOW

Below is a simplified flow of how Meld content is processed:

   ┌─────────────────────────────┐
   │   Meld Source Document      │
   └─────────────────────────────┘
                │
                ▼
   ┌─────────────────────────────┐
   │ ParserService.parse(...)    │
   │   → uses meld-ast to parse  │
   └─────────────────────────────┘
                │ AST (MeldNode[])
                ▼
   ┌─────────────────────────────────────────────────┐
   │ InterpreterService.interpret(nodes, options)   │
   │   → For each node, pass to DirectiveService     │
   └─────────────────────────────────────────────────┘
                │
                ▼
   ┌──────────────────────────────────────────┐
   │ DirectiveService                        │
   │   → For each directive, route to        │
   │     the correct directive handler       │
   └──────────────────────────────────────────┘
                │
                ▼
   ┌───────────────────────────────────────────────┐
   │ StateService + ResolutionService + Others    │
   │   → Where variables are stored/resolved      │
   │   → Path expansions, data lookups, etc.      │
   └───────────────────────────────────────────────┘
                │
                ▼
   ┌──────────────────────────────────────────┐
   │ OutputService (optional)                │
   │   → Convert final AST/State to markdown,│
   │     LLM XML, or other formats.          │
   └──────────────────────────────────────────┘

## MAJOR SERVICES (OVERVIEW)

Below are the key "services" in the codebase. Each follows the single responsibility principle:

### CLIService
   - Provides command-line interface for running Meld
   - Handles file watching and reprocessing
   - Manages format selection and output options
   - Routes to appropriate services based on CLI flags

### ParserService
   - Wraps the meld-ast parse(content) function
   - Adds location information with file paths (parseWithLocations)
   - Produces an array of MeldNode objects

### DirectiveService
   - Routes directives to the correct directive handler
   - Validates directives using ValidationService
   - Calls ResolutionService for variable resolution
   - Updates StateService with directive execution results

### InterpreterService
   - Orchestrates the main interpret(nodes) pipeline
   - For each AST node:
       a) If it's text, store it or pass it along
       b) If it's a directive, calls DirectiveService
   - Maintains the top-level process flow

### StateService
   - Stores variables in maps:
       • textVars (for @text)
       • dataVars (for @data)
       • pathVars (for @path)
       • commands (for @define)
   - Tracks MeldNodes for final structure
   - Provides child states for nested imports
   - Supports immutability toggles

### ResolutionService
   - Handles all variable interpolation:
       • Text variables ("${var}")
       • Data references ("#{data.field}")
       • Path expansions ("$HOMEPATH/path")
       • Command references
   - Context-aware resolution
   - Circular reference detection
   - Sub-fragment parsing support

### CircularityService
   - Prevents infinite import loops
   - Detects circular variable references
   - Maintains dependency graphs

### PathService
   - Validates and normalizes paths
   - Enforces path security constraints
   - Handles path joining and manipulation
   - Supports test mode for path operations

### ValidationService
   - Validates directive syntax and constraints
   - Provides extensible validator registration
   - Throws MeldDirectiveError on validation failures
   - Tracks available directive kinds

###  FileSystemService
    - Abstracts file operations (read, write)
    - Supports both real and test filesystems
    - Handles path resolution and validation

### OutputService
    - Converts final AST and state to desired format
    - Supports markdown and LLM XML output
    - Integrates with llmxml for LLM-friendly formatting
    - Handles format-specific transformations

## TESTING INFRASTRUCTURE

All tests are heavily reliant on a memory-based filesystem (memfs) for isolation and speed. The major testing utilities include:

### MemfsTestFileSystem
   – Thin wrapper around memfs
   – Offers readFile, writeFile, mkdir, etc. with in-memory data
   – Provides an ephemeral environment for all test IO

### TestContext
   – Central test harness that creates a new MemfsTestFileSystem
   – Provides references to all major services (ParserService, DirectiveService, etc.)
   – Allows writing files, snapshotting the FS, and comparing

### TestSnapshot
   – Takes "snapshots" of the current Memfs FS, storing a Map<filePath, content>
   – Compares snapshots to detect added/removed/modified files

### ProjectBuilder
   – Creates mock "projects" in the in-memory FS from JSON structure
   – Useful for complex, multi-file tests or large fixture-based testing

### Node Factories
   – Provides helper functions for creating AST nodes in tests
   – Supports creating directive, text, and code fence nodes
   – Includes location utilities for source mapping

Testing Organization:
• tests/utils/: Core test infrastructure (MemFS, snapshots, contexts)
• tests/mocks/: Minimal mocks and test doubles
• tests/fixtures/: JSON-based test data
• tests/services/: Service-specific integration tests

Testing Approach:
• Each test uses a fresh TestContext or recreates MemfsTestFileSystem
• Direct imports from core packages (meld-ast, meld-spec) for types
• Factory functions for creating test nodes and data
• Snapshots for tracking filesystem changes

## SERVICE RELATIONSHIPS

Below is a more expanded ASCII diagram showing services with references:

                                 +---------------------+
                                 |    CLIService      |
                                 |   Entry point      |
                                 +----------+----------+
                                            |
                                            v
                                 +---------------------+
                                 |    ParserService    |
                                 | meld-ast parsing    |
                                 +----------+----------+
                                            |
                                            v
 +------------+                 +---------------------+
 | Circularity|  <----------->  |  ResolutionService  |
 |  Service   |                 |   Variable/Path     |
 +------------+                 |    Resolution       |
      ^                                   |
      |                                   v
 +------------+  +---------------------+  +-----------+
 | Validation|-> | DirectiveService   |->|StateService|
 |  Service  |   +---------+-----------+  +-----------+
 +------------+            |   |
      ^                    v   v
      |         +---------------+--------------+
      +---------|   Handler(s): text, data,   |
                |   embed, import, etc.       |
                +---------------+--------------+
                                   |
                                   v
                        +---------------------+
                        | InterpreterService |
                        +----------+----------+
                                   |
                                   v
                        +---------------------+
                        |   OutputService    |
                        +---------------------+

Key relationships:
• InterpreterService orchestrates directives → DirectiveService → uses Validation & Resolution.
• ResolutionService consults CircularityService for import cycles, etc.
• DirectiveService updates or reads from StateService.

## EXAMPLE USAGE SCENARIO

1) Input: A .meld file with lines like:
   @text greeting = "Hello"
   @data config = { "value": 123 }
   @import [ path = "other.meld" ]

2) We load the file from disk.
3) ParserService → parse the content → AST.
4) InterpreterService → interpret(AST).
   a) For each directive, DirectiveService → validation → resolution → update StateService.
   b) If an import is encountered, CircularityService ensures no infinite loops.
5) Once done, the final StateService has textVars.greeting = "Hello", dataVars.config = { value: 123 }, etc.
6) OutputService can generate the final text or an LLM-XML representation.

## ERROR HANDLING

• MeldDirectiveError thrown if a directive fails validation or interpretation.
• MeldParseError if the parser cannot parse content.
• PathValidationError for invalid paths.
• ResolutionError for variable resolution issues.
• MeldError as a base class for other specialized errors.

These errors typically bubble up to the caller or test.

## CONCLUSION

This codebase implements the entire Meld language pipeline:
• Parsing Meld documents into an AST.
• Validating & interpreting directives.
• Storing data in a hierarchical state.
• Resolving references (text, data, paths, commands).
• (Optionally) generating final formatted output.

Plus, it has a robust test environment with an in-memory FS, snapshots, and a test harness (TestContext) for integration and unit tests. Everything is layered to keep parsing, state management, directive logic, and resolution separate, adhering to SOLID design principles.

The ASCII diagrams, modules, and file references in this overview represent the CURRENT code as it is: multiple specialized services collaborating to parse and interpret Meld scripts thoroughly—test coverage is facilitated by the in-memory mocking and snapshot-based verification.

\=== TEST SETUP

# Testing in Meld

This document outlines the testing infrastructure and best practices for the Meld codebase. It serves as a practical guide for writing and maintaining tests.

## Directory Structure

Tests are organized following these conventions:

```
project-root/
├─ tests/                    # Test infrastructure and shared resources
│  ├─ utils/                # Test utilities and factories
│  ├─ mocks/                # Shared mock implementations
│  ├─ fixtures/             # Test fixture data
│  └─ setup.ts             # Global test setup
└─ services/               # Service implementations with co-located tests
   └─ ServiceName/
      ├─ ServiceName.test.ts           # Unit tests
      ├─ ServiceName.integration.test.ts # Integration tests
      └─ handlers/
         └─ HandlerName.test.ts        # Handler-specific tests
```

## Test Infrastructure

### Core Testing Utilities

1. **TestContext**
   - Central test harness providing access to all test utilities
   - Manages test state and cleanup
   - Available globally in tests as `testContext`

2. **Test Factories**
   - Located in `tests/utils/testFactories.ts`
   - Provides helper functions for creating test nodes and mocks
   - Ensures consistent test data creation

Example usage:
```typescript
import {
  createDefineDirective,
  createLocation,
  createMockStateService
} from '@tests/utils/testFactories';

const node = createDefineDirective(
  'greet',
  'echo "Hello"',
  [],
  createLocation(1, 1, 1, 20)
);

const mockState = createMockStateService();
```

### Mock Services

The test factories provide mock implementations for all core services:

```typescript
const mockServices = {
  stateService: createMockStateService(),
  validationService: createMockValidationService(),
  resolutionService: createMockResolutionService(),
  fileSystemService: createMockFileSystemService(),
  // ... etc
};
```

Each mock service implements the corresponding interface and provides sensible defaults.

## Writing Tests

### Service Tests

Service tests should follow this structure:

```typescript
describe('ServiceName', () => {
  let service: ServiceName;
  let dependencies: {
    stateService: IStateService;
    // ... other dependencies
  };

  beforeEach(() => {
    dependencies = {
      stateService: createMockStateService(),
      // ... initialize other dependencies
    };
    service = new ServiceName(dependencies);
  });

  describe('core functionality', () => {
    it('should handle basic operations', async () => {
      // Arrange
      const input = // ... test input

      // Act
      const result = await service.operation(input);

      // Assert
      expect(result).toBeDefined();
      expect(dependencies.stateService.someMethod)
        .toHaveBeenCalledWith(expectedArgs);
    });
  });

  describe('error handling', () => {
    it('should handle errors appropriately', async () => {
      // ... error test cases
    });
  });
});
```

### Directive Handler Tests

Directive handler tests should cover:

1. Value Processing
   - Basic value handling
   - Parameter processing
   - Edge cases

2. Validation Integration
   - Integration with ValidationService
   - Validation error handling

3. State Management
   - State updates
   - Command/variable storage

4. Error Handling
   - Validation errors
   - Resolution errors
   - State errors

Example structure:
```typescript
describe('HandlerName', () => {
  let handler: HandlerName;
  let dependencies: {
    validationService: IValidationService;
    stateService: IStateService;
    resolutionService: IResolutionService;
  };

  beforeEach(() => {
    dependencies = {
      validationService: createMockValidationService(),
      stateService: createMockStateService(),
      resolutionService: createMockResolutionService()
    };
    handler = new HandlerName(dependencies);
  });

  describe('value processing', () => {
    // Value processing tests
  });

  describe('validation', () => {
    // Validation tests
  });

  describe('state management', () => {
    // State management tests
  });

  describe('error handling', () => {
    // Error handling tests
  });
});
```

### Integration Tests

Integration tests should focus on real-world scenarios and service interactions:

```typescript
describe('Service Integration', () => {
  let services: {
    directiveService: DirectiveService;
    stateService: StateService;
    // ... other real service instances
  };

  beforeEach(() => {
    services = {
      directiveService: new DirectiveService(),
      stateService: new StateService(),
      // ... initialize other services
    };

    // Initialize service relationships
    services.directiveService.initialize(
      services.validationService,
      services.stateService,
      services.resolutionService
    );
  });

  it('should process complex scenarios', async () => {
    // Test end-to-end flows
  });
});
```

## Best Practices

1. **Test Organization**
   - Co-locate tests with implementation files
   - Use clear, descriptive test names
   - Group related tests using `describe` blocks
   - Follow the Arrange-Act-Assert pattern

2. **Mock Usage**
   - Use the provided mock factories
   - Set up specific mock implementations in beforeEach
   - Clear all mocks between tests
   - Be explicit about mock expectations

3. **Error Testing**
   - Test both expected and unexpected errors
   - Verify error messages and types
   - Test error propagation
   - Include location information in errors

4. **Location Handling**
   - Always include location information in test nodes
   - Use `createLocation` helper for consistency
   - Test location propagation in errors

5. **State Management**
   - Test state immutability
   - Verify state cloning
   - Test parent/child state relationships
   - Validate state updates

## Running Tests

```bash
# Run all tests
npm test

# Run specific test file
npm test services/DirectiveService/handlers/definition/DefineDirectiveHandler.test.ts

# Run tests in watch mode
npm test -- --watch

# Run tests with coverage
npm test -- --coverage
```

## Test Coverage

The project maintains high test coverage through:
- Unit tests for all services and handlers
- Integration tests for service interactions
- Error case coverage
- Edge case testing

Coverage reports can be generated using:
```bash
npm test -- --coverage
```

## Debugging Tests

1. Use the `debug` logger in tests:
```typescript
import { debug } from '@core/utils/logger';

it('should handle complex case', () => {
  debug('Test state:', someObject);
});
```

2. Use Node.js debugger:
   - Add `debugger` statement in test
   - Run `npm test -- --inspect-brk`
   - Connect Chrome DevTools

3. Use Vitest UI:
```bash
npm test -- --ui
```

\=== ISSUES ENCOUNTERED

# Meld Output Processing Issues

## Overview

During the first production run of Meld, we've identified several critical issues with output processing. These issues prevent Meld from correctly processing directives and generating clean output as specified in the grammar.

## Issue 1: Directive Definitions Appearing in Output

### Description
The output currently includes the raw directive definitions themselves instead of just their processed results.

### Expected Behavior
- Plain text/markdown content should appear in output
- Results of 'run' and 'embed' directives should appear in output
- Directive definitions should NOT appear in output
- Definition/import directives (@path, @text, @data, @import, @define) should NOT appear in output
- Comment lines (>>) should NOT appear in output

### Actual Behavior
Looking at example.xml/example.md, we see:
- Directive definitions are being included verbatim
- Directive metadata (kind, identifier, etc.) is being exposed
- XML/MD structure is being created around the directives themselves

### Steps to Reproduce
1. Create a file example.meld with directives
2. Run `meld example.meld`
3. Observe output contains raw directive definitions

### Investigation Notes
- Need to examine OutputService's transformation logic
- Check if AST/State transformation is happening before output generation
- Review how directive results are being stored in state

## Issue 2: Directives Not Being Processed

### Description
The directives' content is not being processed - raw directive content appears instead of processed results.

### Expected Behavior
- @run directives should execute commands and include output
- @embed directives should include file contents
- Variable interpolation should occur
- Results should be properly formatted in output

### Actual Behavior
- Raw directive content appears in output
- Commands are not being executed
- Files are not being embedded
- Variables are not being interpolated

### Steps to Reproduce
1. Create example.meld with @run and @embed directives
2. Run `meld example.meld`
3. Observe raw directives in output instead of processed results

### Investigation Notes
- Need to examine InterpreterService directive processing
- Check DirectiveService handler execution
- Review how results are being stored in State
- Compare with prototype implementation in dev/meld-cli/src

## Issue 3: @embed Variable Input

### Description
The @embed directive is not accepting variables as input, forcing use of @text directives as a workaround.

### Expected Behavior
```meld
@embed [${role_text}]
@embed [#{task.code_review}]
```
Should work as expected, embedding the content referenced by the variables.

### Actual Behavior
Variables in @embed directives are not being processed, requiring workaround:
```meld
@text role_text = `#{role.architect}`
@embed [${role_text}]
```

### Steps to Reproduce
1. Create file with @embed directive using variable input
2. Run meld on the file
3. Observe variable not being processed

### Investigation Notes
- Not a parser limitation (parser supports this functionality)
- Need to examine DirectiveService/handlers for @embed
- Check variable resolution in ResolutionService
- Review how @embed handler processes its input

### Root Cause
After examining the EmbedDirectiveHandler and ResolutionService, the issue appears to be in the variable resolution flow:

1. **EmbedDirectiveHandler Processing**
   ```typescript
   // 1. Get path from directive
   const { path, section } = node.directive;

   // 2. Create resolution context
   const resolutionContext = {
     currentFilePath: context.currentFilePath,
     state: context.state,
     allowedVariableTypes: {
       text: true,
       data: true,
       path: true,
       command: false
     }
   };

   // 3. Resolve variables in path
   const resolvedPath = await this.resolutionService.resolveInContext(
     path,
     resolutionContext
   );
   ```

2. **Resolution Flow**
   - EmbedDirectiveHandler correctly attempts to resolve variables
   - ResolutionService has proper variable resolution support
   - The issue is in the node transformation gap:
     1. Directive is processed and path is resolved
     2. Content is read and parsed
     3. But no node replacement happens
     4. Original directive remains in AST

3. **Variable Resolution Support**
   - ResolutionService supports:
     - Text variables (${var})
     - Data variables (#{data})
     - Path variables ($path)
   - Resolution context allows all variable types
   - Variable resolution itself works correctly

4. **Missing Transformation**
   ```typescript
   // Current flow:
   1. Process directive -> resolve path -> read content
   2. Store in state
   3. Keep original directive node

   // Needed flow:
   1. Process directive -> resolve path -> read content
   2. Create new text/content node
   3. Replace directive node with content node
   ```

### Required Changes for @embed

1. **Node Transformation**
   ```typescript
   class EmbedDirectiveHandler {
     async execute(node: DirectiveNode, context: DirectiveContext): Promise<IStateService> {
       // ... existing resolution code ...

       // Create content node to replace directive
       const contentNode = {
         type: 'Text',
         content: processedContent
       };

       // Replace node in AST
       context.state.replaceNode(node, contentNode);

       return newState;
     }
   }
   ```

2. **State Service Enhancement**
   - Add node replacement capability
   - Track node relationships
   - Support AST modifications

3. **Handler Interface Update**
   ```typescript
   interface IDirectiveHandler {
     execute(node: DirectiveNode, context: DirectiveContext): Promise<{
       state: IStateService;
       replacement?: MeldNode;  // Optional replacement node
     }>;
   }
   ```

4. **Integration Changes**
   - Update InterpreterService to handle node replacements
   - Modify DirectiveService to pass replacements
   - Update OutputService to use transformed nodes

### Testing Strategy

1. **Variable Resolution Tests**
   ```typescript
   it('should handle variable input in embed path', async () => {
     const node = createEmbedDirective('${docPath}', undefined, createLocation(1, 1));
     const context = {
       state: stateService,
       currentFilePath: 'test.meld'
     };

     stateService.getTextVar.mockReturnValue('doc.md');
     fileSystemService.exists.mockResolvedValue(true);
     fileSystemService.readFile.mockResolvedValue('Test content');

     const result = await handler.execute(node, context);
     expect(result.replacement?.type).toBe('Text');
     expect(result.replacement?.content).toBe('Test content');
   });
   ```

2. **Node Replacement Tests**
   ```typescript
   it('should replace directive node with content', async () => {
     const node = createEmbedDirective('doc.md', undefined, createLocation(1, 1));
     const context = { state: stateService };

     const result = await handler.execute(node, context);
     expect(stateService.replaceNode).toHaveBeenCalledWith(
       node,
       expect.objectContaining({ type: 'Text' })
     );
   });
   ```

### Implementation Plan

1. **Phase 1: Node Replacement**
   - Add node replacement to StateService
   - Update handler interface
   - Modify InterpreterService

2. **Phase 2: Variable Resolution**
   - Verify resolution context
   - Add variable resolution tests
   - Update error handling

3. **Phase 3: Integration**
   - Connect with OutputService changes
   - Update pipeline flow
   - Add end-to-end tests

## Prototype Implementation Analysis

The prototype in dev/meld-cli/src takes a fundamentally different approach to processing and output:

### Key Differences

1. **Direct AST Transformation**
   - Uses remark/unified pipeline for markdown processing
   - Transforms nodes in-place during processing
   - Replaces directive nodes with their output content
   - No separate state management or output transformation

2. **Processing Pipeline**
   ```typescript
   unified()
     .use(remarkParse)                  // Parse markdown to AST
     .use(remarkMeldDirectives)         // Identify directives
     .use(remarkProcessMeldNodes)       // Process & replace nodes
     .use(remarkMeldDirectiveHandler)   // Final handling
     .use(remarkStringify)              // Output as markdown
   ```

3. **Node Replacement Strategy**
   - When processing a directive node:
     ```typescript
     parent.children[index] = {
       type: 'html',
       value: processedContent
     } as Node;
     ```
   - Original directive node is completely replaced
   - No trace of directive in final output

4. **Command Execution**
   - Synchronous execution during processing
   - Output captured and inserted directly
   - Both stdout/stderr collected in order
   - ANSI codes stripped from output

5. **Import/Embed Handling**
   - Direct file reading and processing
   - Content immediately inserted into AST
   - Supports both markdown and code files
   - Handles section extraction

### Insights for Current Implementation

1. **AST Transformation**
   - Current implementation may be preserving nodes instead of replacing
   - Need to check if DirectiveService is transforming nodes or just processing them
   - OutputService may be seeing original nodes instead of results

2. **State Management**
   - Prototype has no separate state
   - Our StateService might be storing results but not affecting AST
   - Need to verify how state connects to output generation

3. **Processing Flow**
   - Prototype processes synchronously, top-to-bottom
   - Our pipeline may be deferring execution or storing results separately
   - Need to check if InterpreterService is actually executing commands

4. **Output Generation**
   - Prototype's output is a direct result of AST transformation
   - Our OutputService may need similar node replacement strategy
   - Consider adding pre-output transformation step

### Action Items

1. Check DirectiveService implementation:
   - Are we replacing nodes with their results?
   - How are results being stored?
   - Is AST being modified during processing?

2. Review InterpreterService:
   - Verify command execution timing
   - Check how results are being handled
   - Compare with prototype's direct replacement

3. Examine OutputService:
   - Add pre-output AST transformation
   - Consider adopting prototype's replacement strategy
   - Ensure state results are properly integrated

4. Consider Pipeline Changes:
   - May need additional processing step before output
   - Could add node transformation phase
   - Might need to modify how state affects AST

## Next Steps

1. **Investigation Priority**
   - Issue 1: Output processing (most fundamental)
   - Issue 2: Directive processing
   - Issue 3: @embed variable handling

2. **Investigation Approach**
   - Compare with prototype implementation
   - Review service interactions
   - Add logging/debugging
   - Create minimal test cases

3. **Service Focus Areas**
   - OutputService: Issue 1
   - InterpreterService: Issue 2
   - DirectiveService (@embed handler): Issue 3
   - StateService: All issues (state management)

4. **Questions to Answer**
   - How is the prototype handling output processing differently?
   - Where in the pipeline are directive results being lost?
   - How is state being transformed for output?
   - What assumptions were made during architecture design that need revision?

## Current Implementation Analysis

After examining the current implementation, here are the key findings for each issue:

### Issue 1: Directive Definitions in Output

**Root Cause**: The OutputService's `nodeToMarkdown` method is directly converting directive nodes to markdown without transformation:

```typescript
private async nodeToMarkdown(node: MeldNode, options: OutputOptions): Promise<string> {
  switch (node.type) {
    case 'Directive':
      const directiveNode = node as DirectiveNode;
      // Formats directive as JSON instead of processing its result
      return `### ${directiveNode.directive.kind} Directive\n${JSON.stringify(directiveNode.directive, null, 2)}\n\n`;
    // ...
  }
}
```

This shows that:
1. Directive nodes are being preserved in the AST
2. The OutputService is seeing raw directive nodes
3. No transformation of directives to their results is happening

### Issue 2: Directives Not Being Processed

**Root Cause**: The InterpreterService is storing nodes but not transforming them:

```typescript
switch (node.type) {
  case 'Directive':
    const directiveState = currentState.clone();
    // Just adds the node without transformation
    directiveState.addNode(node);
    currentState = await this.directiveService.processDirective(directiveNode, {
      state: directiveState,
      currentFilePath: state.getCurrentFilePath() ?? undefined
    });
    break;
}
```

The issue is:
1. Directives are processed (by DirectiveService)
2. Results are stored in state
3. But the original node is preserved in the AST
4. No node replacement with results is happening

### Issue 3: @embed Variable Input

**Root Cause**: After examining the EmbedDirectiveHandler and ResolutionService, the issue appears to be in the variable resolution flow:

1. **EmbedDirectiveHandler Processing**
   ```typescript
   // 1. Get path from directive
   const { path, section } = node.directive;

   // 2. Create resolution context
   const resolutionContext = {
     currentFilePath: context.currentFilePath,
     state: context.state,
     allowedVariableTypes: {
       text: true,
       data: true,
       path: true,
       command: false
     }
   };

   // 3. Resolve variables in path
   const resolvedPath = await this.resolutionService.resolveInContext(
     path,
     resolutionContext
   );
   ```

2. **Resolution Flow**
   - EmbedDirectiveHandler correctly attempts to resolve variables
   - ResolutionService has proper variable resolution support
   - The issue is in the node transformation gap:
     1. Directive is processed and path is resolved
     2. Content is read and parsed
     3. But no node replacement happens
     4. Original directive remains in AST

3. **Variable Resolution Support**
   - ResolutionService supports:
     - Text variables (${var})
     - Data variables (#{data})
     - Path variables ($path)
   - Resolution context allows all variable types
   - Variable resolution itself works correctly

4. **Missing Transformation**
   ```typescript
   // Current flow:
   1. Process directive -> resolve path -> read content
   2. Store in state
   3. Keep original directive node

   // Needed flow:
   1. Process directive -> resolve path -> read content
   2. Create new text/content node
   3. Replace directive node with content node
   ```

### Required Changes for @embed

1. **Node Transformation**
   ```typescript
   class EmbedDirectiveHandler {
     async execute(node: DirectiveNode, context: DirectiveContext): Promise<IStateService> {
       // ... existing resolution code ...

       // Create content node to replace directive
       const contentNode = {
         type: 'Text',
         content: processedContent
       };

       // Replace node in AST
       context.state.replaceNode(node, contentNode);

       return newState;
     }
   }
   ```

2. **State Service Enhancement**
   - Add node replacement capability
   - Track node relationships
   - Support AST modifications

3. **Handler Interface Update**
   ```typescript
   interface IDirectiveHandler {
     execute(node: DirectiveNode, context: DirectiveContext): Promise<{
       state: IStateService;
       replacement?: MeldNode;  // Optional replacement node
     }>;
   }
   ```

4. **Integration Changes**
   - Update InterpreterService to handle node replacements
   - Modify DirectiveService to pass replacements
   - Update OutputService to use transformed nodes

### Testing Strategy

1. **Variable Resolution Tests**
   ```typescript
   it('should handle variable input in embed path', async () => {
     const node = createEmbedDirective('${docPath}', undefined, createLocation(1, 1));
     const context = {
       state: stateService,
       currentFilePath: 'test.meld'
     };

     stateService.getTextVar.mockReturnValue('doc.md');
     fileSystemService.exists.mockResolvedValue(true);
     fileSystemService.readFile.mockResolvedValue('Test content');

     const result = await handler.execute(node, context);
     expect(result.replacement?.type).toBe('Text');
     expect(result.replacement?.content).toBe('Test content');
   });
   ```

2. **Node Replacement Tests**
   ```typescript
   it('should replace directive node with content', async () => {
     const node = createEmbedDirective('doc.md', undefined, createLocation(1, 1));
     const context = { state: stateService };

     const result = await handler.execute(node, context);
     expect(stateService.replaceNode).toHaveBeenCalledWith(
       node,
       expect.objectContaining({ type: 'Text' })
     );
   });
   ```

### Implementation Plan

1. **Phase 1: Node Replacement**
   - Add node replacement to StateService
   - Update handler interface
   - Modify InterpreterService

2. **Phase 2: Variable Resolution**
   - Verify resolution context
   - Add variable resolution tests
   - Update error handling

3. **Phase 3: Integration**
   - Connect with OutputService changes
   - Update pipeline flow
   - Add end-to-end tests

### Architecture Gap

The key architectural difference from the prototype:

1. **State vs AST**
   - Prototype: Directly modifies AST, replacing nodes with results
   - Current: Stores results in state but preserves original AST

2. **Processing Flow**
   - Prototype: Immediate node replacement during processing
   - Current: Two-phase approach (process then output) without transformation

3. **Output Generation**
   - Prototype: Simply stringifies transformed AST
   - Current: Tries to handle both AST and state, but only uses AST

### Required Changes

1. **Node Transformation**
   - Add node transformation phase in InterpreterService
   - Replace directive nodes with their results
   - Keep state for variable tracking only

2. **Output Processing**
   - Modify OutputService to handle transformed nodes
   - Remove directive-specific output formatting
   - Use state only for variable resolution

3. **Handler Updates**
   - Update handlers to return result nodes
   - Modify DirectiveService to handle node replacement
   - Ensure EmbedDirectiveHandler properly resolves variables

4. **Pipeline Modification**
   ```typescript
   // Current flow:
   parse -> interpret -> store in state -> output raw nodes

   // Needed flow:
   parse -> interpret -> transform nodes -> output transformed nodes
   ```

### Next Investigation Steps

1. **EmbedDirectiveHandler**
   - Examine implementation
   - Check variable resolution
   - Verify path handling

2. **Node Transformation**
   - Design node replacement strategy
   - Identify transformation point
   - Plan handler modifications

3. **State Management**
   - Review state usage
   - Determine what stays in state
   - Plan state/AST separation

\=== COMPLETED PLAN FOR ADDRESSING ISSUES

# Implementation Plan

## Issues

During the first production run of Meld, we've identified several critical issues with output processing. These issues prevent Meld from correctly processing directives and generating clean output as specified in the grammar.

## Issue 1: Directive Definitions Appearing in Output ⏳ (In Progress)

### Description
The output currently includes the raw directive definitions themselves instead of just their processed results.

### Expected Behavior
- Plain text/markdown content should appear in output
- Results of 'run' and 'embed' directives should appear in output
- Directive definitions should NOT appear in output
- Definition/import directives (@path, @text, @data, @import, @define) should NOT appear in output
- Comment lines (>>) should NOT appear in output

### Actual Behavior
Looking at example.xml/example.md, we see:
- Directive definitions are being included verbatim
- Directive metadata (kind, identifier, etc.) is being exposed
- XML/MD structure is being created around the directives themselves

## Issue 2: Directives Not Being Processed ⏳ (In Progress)

### Description
The directives' content is not being processed - raw directive content appears instead of processed results.

### Expected Behavior
- @run directives should execute commands and include output
- @embed directives should include file contents
- Variable interpolation should occur
- Results should be properly formatted in output

### Actual Behavior
- Raw directive content appears in output
- Commands are not being executed
- Files are not being embedded
- Variables are not being interpolated

## Issue 3: @embed Variable Input (Pending)

### Description
The @embed directive is not accepting variables as input, forcing use of @text directives as a workaround.

### Expected Behavior
```meld
@embed [${role_text}]
@embed [#{task.code_review}]
```
Should work as expected, embedding the content referenced by the variables.

### Actual Behavior
Variables in @embed directives are not being processed, requiring workaround:
```meld
@text role_text = `#{role.architect}`
@embed [${role_text}]
```

---

# Additional Context and Constraints

### Path Handling
- Currently using enhanced PathService for all path-related functionality
- Path resolution is consistent across @import, @embed, and @path directives
- Security constraints are maintained through PathService
- Path validation happens in PathService
- See [dev/PATHS.md] for more detail

### Testing Infrastructure
- Tests co-located with implementation files
- TestContext provides central test harness
- Mock services available through test factories
- High test coverage (484 passing tests)
- See [docs/TESTS.md] for more detail

---

# PLAN FOR ADDRESSING ISSUES

## Incremental Implementation Strategy

### Phase 1: Add New Functionality Without Breaking Existing ✅ (Completed)

1. **StateService Enhancement (Step 1)** ✅
- Added transformation support to StateNode interface
- Implemented transformation methods in StateService
- Added comprehensive tests for transformation functionality
- All tests passing

2. **DirectiveHandler Interface Update (Step 2)** ✅
- Added DirectiveResult interface with replacement node support
- Updated base handler implementation
- Maintained backward compatibility

3. **InterpreterService Feature Flag (Step 3)** ✅
- Added transformation feature flag
- Implemented node transformation support
- Maintained existing behavior when disabled

### Phase 2: Gradual Handler Migration ✅ (Completed)

1. **EmbedDirectiveHandler Migration** ✅ (Completed)
   - Update to support node replacement
   - Maintain path handling through PathService
   - Add transformation tests
   - Verify both with feature flag on/off

2. **RunDirectiveHandler Migration** ✅ (Completed)
   - Similar process to EmbedDirectiveHandler
   - Focus on command execution results
   - Add transformation tests
   - Verify both behaviors

3. **Other Handlers** ✅ (Completed)
   - ImportDirectiveHandler successfully migrated
   - Added handler-specific transformation tests
   - Verified behavior in both modes

### Phase 3: OutputService Update (Pending)

1. **Add Dual-Mode Support**
```typescript
class OutputService {
    async convert(state: IStateService, format: OutputFormat): Promise<string> {
        const nodes = this.useNewTransformation
            ? state.getTransformedNodes()
            : state.getNodes();
        return this.nodesToFormat(nodes, format);
    }
}
```

2. **Update Tests**
   - Add transformation-aware tests
   - Verify output with both modes
   - Test complex scenarios

### Phase 3 Implementation Notes

1. **Key Learnings from Handler Migration**
   - All handlers now support both modes through `isTransformationEnabled()`
   - Transformed nodes preserve original location for error reporting
   - Each handler type has specific transformation behavior:
     - EmbedDirectiveHandler: Replaces with embedded content
     - RunDirectiveHandler: Replaces with command output
     - ImportDirectiveHandler: Removes from output (empty text node)

2. **State Management Insights**
   - Transformation state is tracked via `isTransformationEnabled()`
   - State cloning preserves transformation status
   - Child states inherit transformation mode
   - All state mutations maintain immutability

3. **Testing Strategy for OutputService**
   - Create separate transformation test file
   - Test each directive type's output behavior
   - Verify complex documents with mixed content
   - Test error cases in both modes
   - Ensure proper cleanup of directive metadata

4. **Potential Challenges**
   - Handling mixed content (directives + text)
   - Preserving formatting and whitespace
   - Managing directive-specific output rules
   - Error reporting with transformed nodes

5. **Success Criteria for Phase 3**
   - No directive definitions in output
   - Clean, properly formatted content
   - Correct handling of all directive types
   - Proper error reporting
   - Backward compatibility maintained

### Phase 4: Cleanup (Pending)

Once all handlers are migrated and tests pass:

1. Remove feature flags
2. Remove old state tracking
3. Update documentation
4. Clean up tests

## Testing Strategy

1. **Isolation**
   - New tests in *.transformation.test.ts files
   - Use TestContext for consistent setup
   - Leverage existing mock services

2. **Verification Points**
   - After each step, run full test suite
   - Verify both old and new behavior
   - Check path handling remains correct
   - Validate security constraints

3. **Coverage**
   - Maintain existing test coverage
   - Add transformation-specific cases
   - Test edge cases in both modes

## Rollback Plan

Each phase can be rolled back independently:
1. Feature flags allow quick behavior switches
2. Separate test files ease removal
3. Dual-mode implementation provides fallback

## Success Criteria

1. All 484 existing tests continue to pass
2. New transformation tests pass
3. Path handling remains secure and consistent
4. No regression in existing functionality
5. Clean separation of concerns maintained

## Test Coverage Analysis

### Existing Tests

1. **State Management** ✅
   - Variable storage and retrieval
   - Command definitions
   - State inheritance
   - State cloning

2. **Basic Directive Validation** ✅
   - Syntax validation
   - Required fields
   - Type checking

3. **Path Handling** ✅
   - Path resolution
   - Path validation
   - Directory handling

4. **Import Management** ✅
   - Circular import detection
   - Import scope
   - File resolution

### Tests Needing Changes

1. **OutputService Tests**
   ```typescript
   // Current:
   it('should convert directive nodes to markdown', async () => {
     const nodes = [createDirectiveNode('test', { value: 'example' })];
     const output = await service.convert(nodes, state, 'markdown');
     expect(output).toContain('### test Directive');
   });

   // Needed:
   it('should output directive results not definitions', async () => {
     const nodes = [createDirectiveNode('run', { command: 'echo test' })];
     const output = await service.convert(nodes, state, 'markdown');
     expect(output).toBe('test\n');
   });
   ```

2. **EmbedDirectiveHandler Tests**
   ```typescript
   // Current:
   it('should handle basic embed without modifiers', async () => {
     // Tests state updates but not node replacement
   });

   // Needed:
   it('should replace embed directive with file contents', async () => {
     const node = createEmbedDirective('test.md');
     const result = await handler.execute(node, context);
     expect(result.replacement.type).toBe('Text');
     expect(result.replacement.content).toBe('file contents');
   });
   ```

3. **RunDirectiveHandler Tests**
   ```typescript
   // Needed:
   it('should replace run directive with command output', async () => {
     const node = createRunDirective('echo test');
     const result = await handler.execute(node, context);
     expect(result.replacement.type).toBe('Text');
     expect(result.replacement.content).toBe('test\n');
   });

   it('should timeout long-running commands', async () => {
     const node = createRunDirective('sleep 1000');
     await expect(handler.execute(node, context))
       .rejects.toThrow('Command timed out');
   });
   ```

### New Tests Needed

1. **AST Transformation Tests**
   ```typescript
   describe('AST Transformation', () => {
     it('should transform directive nodes to result nodes', async () => {
       const ast = [
         createTextNode('before\n'),
         createRunDirective('echo test'),
         createTextNode('after\n')
       ];
       const result = await inte