meld/_dev/prompts/old/debug-tool-context.md

Meld: A template language for LLM prompts

# REQUIRED READING:

## === KEY DOCS ====================

# Meld Architecture

## IMPORTANT

Meld has an incredibly robust architecture.

KEY PRINCIPLE: If something ideally _should_ be abstracted out in a clean way in an ideal case, it likely _already is_. So don't assume it's not and reimplement work and add complexity that might have already be handled by another part of the codebase.

## INTRODUCTION

Meld is a specialized, directive-based scripting language designed for embedding small "@directives" inside an otherwise plain text (e.g., Markdown-like) document. The code in this repository implements:

• Meld grammar rules and token types (e.g., text directives, path directives, data directives).
• The parsing layer that converts Meld content into an AST (Abstract Syntax Tree).
• A directive interpretation layer that processes these AST nodes and manipulates internal "states" to store variables and more.
• A resolution layer to handle variable references, path expansions, data manipulations, etc.
• Testing utilities and an in-memory FS (memfs) to simulate filesystems for thorough testing.

The main idea:
1. Meld code is parsed to an AST.
2. Each directive node is validated and interpreted, updating a shared "state" (variables, data structures, commands, etc.).
3. Optional transformations (e.g., output formatting) generate final representations (Markdown, LLM-friendly XML, etc.).

Below is an overview of the directory and service-level architecture, referencing code from this codebase.

## DIRECTORY & FILE STRUCTURE

At a high level, the project is arranged as follows (select key entries included):

project-root/
 ├─ api/                    ← High-level API and tests
 │   ├─ api.test.ts
 │   └─ index.ts
 ├─ bin/                    ← CLI entry point
 │   └─ meld.ts
 ├─ cli/                    ← CLI implementation
 │   ├─ cli.test.ts
 │   └─ index.ts
 ├─ core/                   ← Core utilities and types
 │   ├─ config/            ← Configuration (logging, etc.)
 │   ├─ errors/            ← Error class definitions
 │   │   ├─ MeldError.ts
 │   │   ├─ ServiceInitializationError.ts   ← Service initialization errors
 │   │   └─ ... other errors
 │   ├─ types/             ← Core type definitions
 │   │   ├─ dependencies.ts  ← Service dependency definitions
 │   │   └─ index.ts
 │   └─ utils/             ← Logging and utility modules
 │       ├─ logger.ts
 │       ├─ serviceValidation.ts  ← Service validation utilities
 │       └─ simpleLogger.ts
 ├─ services/              ← Core service implementations
 │   ├─ pipeline/          ← Main transformation pipeline
 │   │   ├─ ParserService/     ← Initial parsing
 │   │   ├─ InterpreterService/← Pipeline orchestration
 │   │   ├─ DirectiveService/  ← Directive handling
 │   │   │   ├─ handlers/
 │   │   │   │   ├─ definition/   ← Handlers for definition directives
 │   │   │   │   └─ execution/    ← Handlers for execution directives
 │   │   │   └─ errors/
 │   │   └─ OutputService/    ← Final output generation
 │   ├─ state/             ← State management
 │   │   ├─ StateService/      ← Core state management
 │   │   └─ StateEventService/ ← Core event system
 │   ├─ resolution/        ← Resolution and validation
 │   │   ├─ ResolutionService/ ← Variable/path resolution
 │   │   ├─ ValidationService/ ← Directive validation
 │   │   └─ CircularityService/← Circular dependency detection
 │   ├─ fs/                ← File system operations
 │   │   ├─ FileSystemService/ ← File operations
 │   │   ├─ PathService/      ← Path handling
 │   │   └─ PathOperationsService/ ← Path utilities
 │   └─ cli/               ← Command line interface
 │       └─ CLIService/    ← CLI entry point
 ├─ tests/                  ← Test infrastructure
 │   ├─ fixtures/          ← Test fixture data
 │   ├─ mocks/             ← Test mock implementations
 │   └─ utils/             ← Test utilities and helpers
 │       ├─ debug/         ← Test debug utilities
 │       │   ├─ StateDebuggerService/
 │       │   ├─ StateVisualizationService/
 │       │   ├─ StateHistoryService/
 │       │   └─ StateTrackingService/
 │       ├─ FixtureManager.ts
 │       ├─ MemfsTestFileSystem.ts
 │       ├─ ProjectBuilder.ts
 │       ├─ TestContext.ts
 │       └─ TestSnapshot.ts
 ├─ docs/                   ← Documentation
 ├─ package.json
 ├─ tsconfig.json
 ├─ tsup.config.ts
 └─ vitest.config.ts

Key subfolders:
• services/pipeline/: Core transformation pipeline services (parsing, interpretation, directives, output)
• services/state/: State management and event services
• services/resolution/: Resolution, validation, and circularity detection services
• services/fs/: File system, path handling, and operations services
• services/cli/: Command line interface services
• core/: Central types, errors, and utilities used throughout the codebase
• tests/utils/: Test infrastructure including debug utilities, memfs implementation, fixture management, and test helpers
• api/: High-level public API for using Meld programmatically
• cli/: Command line interface for Meld

## CORE LIBRARIES & THEIR ROLE

### meld-ast
   • parse(content: string): MeldNode[]
   • Basic parsing that identifies directives vs. text nodes.
   • Produces an AST which other services manipulate.

### llmxml
   • Converts content to an LLM-friendly XML format or can parse partially.
   • OutputService may call it if user requests "llm" format.

### meld-spec
   • Contains interface definitions for MeldNode, DirectiveNode, TextNode, etc.
   • Contains directive kind enumerations.

## HIGH-LEVEL FLOW

Below is a simplified flow of how Meld content is processed:

   ┌─────────────────────────────┐
   │   Meld Source Document      │
   └─────────────────────────────┘
                │
                ▼
   ┌─────────────────────────────┐
   │ ParserService.parse(...)    │
   │   → uses meld-ast to parse  │
   └─────────────────────────────┘
                │ AST (MeldNode[])
                ▼
   ┌─────────────────────────────────────────────────┐
   │ InterpreterService.interpret(nodes, options)    │
   │   → For each node, pass to DirectiveService     │
   │   → Handles node transformations                │
   └─────────────────────────────────────────────────┘
                │
                ▼
   ┌──────────────────────────────────────────┐
   │ DirectiveService                         │
   │   → Routes to correct directive handler  │
   │   → Handlers can provide replacements    │
   └──────────────────────────────────────────┘
                │
                ▼
   ┌───────────────────────────────────────────────┐
   │ StateService + ResolutionService + Others     │
   │   → Stores variables and transformed nodes    │
   │   → Path expansions, data lookups, etc.       │
   └───────────────────────────────────────────────┘
                │
                ▼
   ┌──────────────────────────────────────────┐
   │ OutputService                            │
   │   → Uses transformed nodes for output    │
   │   → Generates clean, directive-free      │
   │     markdown, LLM XML, or other formats  │
   └──────────────────────────────────────────┘

## MAJOR SERVICES (OVERVIEW)

Below are the key "services" in the codebase. Each follows the single responsibility principle:

### CLIService
   - Provides command-line interface for running Meld
   - Handles file watching and reprocessing
   - Manages format selection and output options
   - Routes to appropriate services based on CLI flags

### ParserService
   - Wraps the meld-ast parse(content) function
   - Adds location information with file paths (parseWithLocations)
   - Produces an array of MeldNode objects

### DirectiveService
   - Routes directives to the correct directive handler
   - Validates directives using ValidationService
   - Calls ResolutionService for variable resolution
   - Updates StateService with directive execution results
   - Supports node transformation through DirectiveResult interface
   - Handlers can provide replacement nodes for transformed output

### InterpreterService
   - Orchestrates the main interpret(nodes) pipeline
   - For each AST node:
       a) If it's text, store it or pass it along
       b) If it's a directive:
          - Calls DirectiveService for processing
          - Handles node transformations if provided
          - Updates state with transformed nodes
   - Maintains the top-level process flow
   - Supports transformation mode through feature flags

### StateService
   - Stores variables in maps:
       • textVars (for @text)
       • dataVars (for @data)
       • pathVars (for @path)
       • commands (for @define)
   - Tracks both original and transformed MeldNodes
   - Provides transformation capabilities for directive processing
   - Maintains transformation state during cloning
   - Provides child states for nested imports
   - Supports immutability toggles

### ResolutionService
   - Handles all variable interpolation:
       • Variables ("{{var}}", "{{data.field}}")
       • Path expansions ("$HOMEPATH/path")
       • Command references
   - Context-aware resolution
   - Circular reference detection
   - Sub-fragment parsing support

### CircularityService
   - Prevents infinite import loops
   - Detects circular variable references
   - Maintains dependency graphs

### PathService
   - Validates and normalizes paths
   - Enforces path security constraints
   - Handles path joining and manipulation
   - Supports test mode for path operations

### ValidationService
   - Validates directive syntax and constraints
   - Provides extensible validator registration
   - Throws MeldDirectiveError on validation failures
   - Tracks available directive kinds

###  FileSystemService
    - Abstracts file operations (read, write)
    - Supports both real and test filesystems
    - Handles path resolution and validation

### OutputService
    - Converts final AST and state to desired format
    - Uses transformed nodes when available
    - Supports markdown and LLM XML output
    - Integrates with llmxml for LLM-friendly formatting
    - Handles format-specific transformations
    - Provides clean output without directive definitions

## TESTING INFRASTRUCTURE

All tests are heavily reliant on a memory-based filesystem (memfs) for isolation and speed. The major testing utilities include:

### MemfsTestFileSystem
   – Thin wrapper around memfs
   – Offers readFile, writeFile, mkdir, etc. with in-memory data
   – Provides an ephemeral environment for all test IO

### TestContext
   – Central test harness that creates a new MemfsTestFileSystem
   – Provides references to all major services (ParserService, DirectiveService, etc.)
   – Allows writing files, snapshotting the FS, and comparing

### TestSnapshot
   – Takes "snapshots" of the current Memfs FS, storing a Map<filePath, content>
   – Compares snapshots to detect added/removed/modified files

### ProjectBuilder
   – Creates mock "projects" in the in-memory FS from JSON structure
   – Useful for complex, multi-file tests or large fixture-based testing

### Node Factories
   – Provides helper functions for creating AST nodes in tests
   – Supports creating directive, text, and code fence nodes
   – Includes location utilities for source mapping

Testing Organization:
• tests/utils/: Core test infrastructure (MemFS, snapshots, contexts)
• tests/mocks/: Minimal mocks and test doubles
• tests/fixtures/: JSON-based test data
• tests/services/: Service-specific integration tests

Testing Approach:
• Each test uses a fresh TestContext or recreates MemfsTestFileSystem
• Direct imports from core packages (meld-ast, meld-spec) for types
• Factory functions for creating test nodes and data
• Snapshots for tracking filesystem changes

## DEBUGGING INFRASTRUCTURE

The codebase includes specialized debugging services located in `tests/utils/debug/` that help diagnose and troubleshoot state-related issues:

### StateDebuggerService
   - Provides debug session management and diagnostics
   - Tracks state operations and transformations
   - Offers operation tracing and analysis
   - Helps identify state manipulation issues

### StateVisualizationService
   - Generates visual representations of state
   - Creates Mermaid/DOT graphs of state relationships
   - Visualizes state metrics and transformations
   - Aids in understanding complex state changes

### StateHistoryService
   - Records chronological state changes
   - Maintains operation history
   - Tracks transformation chains
   - Enables state change replay and analysis

### StateTrackingService
   - Monitors state relationships and dependencies
   - Tracks state lineage and inheritance
   - Records metadata about state changes
   - Helps debug scope and inheritance issues

Debugging Approach:
• Services can be enabled selectively in tests
• Debug output includes detailed state snapshots
• Visual representations help understand complex states
• History tracking enables step-by-step analysis

These debugging services are particularly useful for:
• Troubleshooting complex state transformations
• Understanding directive processing chains
• Analyzing variable resolution paths
• Debugging scope inheritance issues
• Visualizing state relationships

## SERVICE RELATIONSHIPS

Services in Meld follow a strict initialization order and dependency graph:

1. Base Services:
   - FileSystemService (no dependencies)
   - PathService (depends on FS)

2. State Management:
   - StateEventService (no dependencies)
   - StateService (depends on events)

3. Core Pipeline:
   - ParserService (independent)
   - ResolutionService (depends on State, FS)
   - ValidationService (depends on Resolution)
   - CircularityService (depends on Resolution)

4. Pipeline Orchestration:
   - DirectiveService (depends on multiple services)
   - InterpreterService (orchestrates others)

5. Output Generation:
   - OutputService (depends on State)

6. Debug Support:
   - DebuggerService (optional, depends on all)

Service initialization and validation is handled through the core/types/dependencies.ts system,
which ensures services are created in the correct order and all dependencies are satisfied.

## EXAMPLE USAGE SCENARIO

1) Input: A .meld file with lines like:
   @text greeting = "Hello"
   @data config = { "value": 123 }
   @import [ path = "other.meld" ]

2) We load the file from disk.
3) ParserService → parse the content → AST.
4) InterpreterService → interpret(AST).
   a) For each directive, DirectiveService → validation → resolution → update StateService.
   b) If an import is encountered, CircularityService ensures no infinite loops.
5) Once done, the final StateService has textVars.greeting = "Hello", dataVars.config = { value: 123 }, etc.
6) OutputService can generate the final text or an LLM-XML representation.

## ERROR HANDLING

• MeldDirectiveError thrown if a directive fails validation or interpretation.
• MeldParseError if the parser cannot parse content.
• PathValidationError for invalid paths.
• ResolutionError for variable resolution issues.
• MeldError as a base class for other specialized errors.

These errors typically bubble up to the caller or test.

## CONCLUSION

This codebase implements the entire Meld language pipeline:
• Parsing Meld documents into an AST.
• Validating & interpreting directives.
• Storing data in a hierarchical state.
• Resolving references (text, data, paths, commands).
• (Optionally) generating final formatted output.

Plus, it has a robust test environment with an in-memory FS, snapshots, and a test harness (TestContext) for integration and unit tests. Everything is layered to keep parsing, state management, directive logic, and resolution separate, adhering to SOLID design principles.

The ASCII diagrams, modules, and file references in this overview represent the CURRENT code as it is: multiple specialized services collaborating to parse and interpret Meld scripts thoroughly—test coverage is facilitated by the in-memory mocking and snapshot-based verification.

# Meld Pipeline Flow

## Overview

The Meld pipeline processes `.meld` files through several stages to produce either `.xml` or `.md` output. Here's a detailed look at how it works:

```ascii
┌─────────────┐     ┌─────────────┐     ┌──────────────┐     ┌──────────────┐
│  Service    │     │   Service   │     │   Pipeline   │     │    Final     │
│Initialization├────►│ Validation  ├────►│  Execution   ├────►│   Output     │
└─────────────┘     └─────────────┘     └──────────────┘     └──────────────┘
      │                    │                    │                    │
      ▼                    ▼                    ▼                    ▼
┌─────────────┐     ┌─────────────┐     ┌──────────────┐     ┌──────────────┐
│Dependencies │     │Validate All │     │Process Input │     │Generate Clean│
│  Resolved   │     │ Services    │     │   Content    │     │   Output    │
└─────────────┘     └─────────────┘     └──────────────┘     └──────────────┘
```

## Service Organization

The pipeline is organized into logical service groups, with strict initialization order and dependency validation:

### Pipeline Services (services/pipeline/)
```ascii
┌─────────────┐     ┌─────────────┐     ┌──────────────┐     ┌──────────────┐
│   Parser    │     │  Directive  │     │ Interpreter  │     │   Output     │
│   Service   ├────►│   Service   ├────►│   Service    ├────►│   Service    │
└─────────────┘     └─────────────┘     └──────────────┘     └──────────────┘
      │                    │                    │                    │
      ▼                    ▼                    ▼                    ▼
┌─────────────┐     ┌─────────────┐     ┌──────────────┐     ┌──────────────┐
│Initialize & │     │Validate &   │     │Transform &   │     │Format &     │
│  Validate   │     │Process Dirs │     │Update State  │     │Generate Out │
└─────────────┘     └─────────────┘     └──────────────┘     └──────────────┘
```

### State Services (services/state/)
```ascii
┌─────────────┐     ┌─────────────┐
│    State    │     │    State    │
│   Service   ├────►│    Event    │
└─────────────┘     │   Service   │
                    └─────────────┘
```

### Resolution Services (services/resolution/)
```ascii
┌─────────────┐     ┌─────────────┐     ┌──────────────┐
│ Resolution  │     │ Validation  │     │ Circularity  │
│   Service   ├────►│   Service   ├────►│   Service    │
└─────────────┘     └─────────────┘     └──────────────┘
```

### File System Services (services/fs/)
```ascii
┌─────────────┐     ┌─────────────┐     ┌──────────────┐
│    File     │     │    Path     │     │     Path     │
│   System    ├────►│   Service   ├────►│  Operations  │
│   Service   │     │             │     │   Service    │
└─────────────┘     └─────────────┘     └──────────────┘
```

## Detailed Flow

1. **Service Initialization** (`core/types/dependencies.ts`)
   ```ascii
   ┌─────────────┐
   │Load Service │
   │Dependencies │
   └─────┬───────┘
         │
         ▼
   ┌─────────────┐
   │Initialize in│
   │   Order    │
   └─────┬───────┘
         │
         ▼
   ┌─────────────┐
   │  Validate   │
   │  Services   │
   └─────────────┘
   ```
   - Resolves service dependencies
   - Initializes in correct order
   - Validates service configuration
   - Enables transformation if requested

2. **Input Processing** (`CLIService`)
   - User runs `meld prompt.meld`
   - `CLIService` handles command line options
   - Default output is `.xml` format
   - Can specify `--format markdown` for `.md` output
   - Supports `--stdout` for direct console output

3. **Parsing** (`ParserService`)
   ```ascii
   ┌─────────────┐
   │  Raw Text   │
   │   Input     │
   └─────┬───────┘
         │
         ▼
   ┌─────────────┐
   │  meld-ast   │
   │   Parser    │
   └─────┬───────┘
         │
         ▼
   ┌─────────────┐
   │ MeldNode[]  │
   │    AST      │
   └─────────────┘
   ```
   - Reads the input file content
   - Parses into AST using `meld-ast`
   - Identifies directives and text nodes
   - Adds source location information

4. **Interpretation** (`InterpreterService`)
   ```ascii
   ┌─────────────┐     ┌─────────────┐
   │  MeldNode[] │     │  Directive  │
   │     AST     ├────►│   Service   │
   └─────────────┘     └──────┬──────┘
                              │
                              ▼
   ┌─────────────┐     ┌─────────────┐
   │ Resolution  │◄────┤   Handler   │
   │   Service   │     │(with node   │
   └──────┬──────┘     │replacements)│
          │            └─────────────┘
          ▼
   ┌─────────────┐
   │    State    │
   │   Service   │
   │(Original &  │
   │Transformed) │
   └─────────────┘
   ```
   - Processes each AST node sequentially
   - Routes directives to appropriate handlers
   - Handlers can provide replacement nodes
   - Maintains both original and transformed states
   - Resolves variables and references
   - Handles file imports and embedding

5. **Output Generation** (`OutputService`)
   ```ascii
   ┌─────────────┐     ┌─────────────┐
   │Transformed  │     │   Format    │
   │  Nodes &    ├────►│  Converter  │
   │   State     │     └──────┬──────┘
                              │
                              ▼
   ┌─────────────┐     ┌─────────────┐
   │Clean Output │◄────┤  Formatted  │
   │(No Directive│     │   Output    │
   │Definitions) │     └─────────────┘
   └─────────────┘
   ```
   - Takes transformed nodes and state
   - Converts to requested format:
     - `llm`: Uses `llmxml` library for LLM-friendly XML
     - `markdown`: Clean markdown without directive definitions
   - Writes output to file or stdout

## Transformation Mode and Variable Resolution

When transformation mode is enabled, the pipeline handles directives and variables in a special way. Understanding this flow is critical for debugging and enhancing directive handlers:

```ascii
┌─────────────┐     ┌─────────────┐     ┌──────────────┐     ┌──────────────┐
│  Directive  │     │Interpretation│     │   State      │     │   Output     │
│  Handlers   ├────►│  & Node     ├────►│  Variable    ├────►│  Generation  │
│(with replace│     │Transformation│     │  Resolution  │     │              │
│  nodes)     │     │              │     │              │     │              │
└─────────────┘     └─────────────┘     └──────────────┘     └──────────────┘
```

### Key Transformation Pipeline Concepts

1. **Directive Handler Replacement Nodes**
   - Directive handlers can return replacement nodes when in transformation mode
   - The InterpreterService must properly apply these replacements in the transformed nodes array
   - For import directives, the replacement is typically an empty text node
   - For embed directives, the replacement node contains the embedded content

2. **State Propagation Across Boundaries**
   - Variables must be explicitly copied between parent and child states
   - When importing files, variables must be copied from imported state to parent state
   - The ImportDirectiveHandler must ensure all variable types (text, data, path, commands) are copied

3. **Variable Resolution Process**
   - Variables can be resolved at multiple stages:
     - During directive processing
     - During node transformation
     - During final output generation
     - During post-processing in the main function
   - The OutputService's nodeToMarkdown method handles variable reference resolution in text nodes
   - A final variable resolution pass in the main function ensures any remaining references are resolved

4. **State Management for Transformation**
   - The StateService maintains both original and transformed node arrays
   - Transformed nodes must be explicitly initialized
   - The transformNode method is used to replace directive nodes with their outputs
   - State must keep track of transformation options to determine which directives to transform

## Service Responsibilities

### Pipeline Services

1. **ParserService** (`services/pipeline/ParserService/`)
   - Wraps meld-ast parser
   - Produces AST nodes
   - Adds file location information

2. **InterpreterService** (`services/pipeline/InterpreterService/`)
   - Orchestrates directive processing
   - Handles node transformations
   - Maintains interpretation state
   - Handles imports and embedding
   - **Critical for transformation:** Applies directive handler replacement nodes to transformed node array
   - **State propagation:** Ensures proper variable inheritance between parent and child states

3. **DirectiveService** (`services/pipeline/DirectiveService/`)
   - Routes directives to handlers
   - Validates directive syntax
   - Supports node transformation
   - Updates state based on directive results
   - **Directive handlers:** Can return replacement nodes in transformation mode
   - **Handler context:** Includes parent state for proper variable propagation

4. **OutputService** (`services/pipeline/OutputService/`)
   - Uses transformed nodes for clean output
   - Supports markdown and LLM XML
   - Generates directive-free output
   - Handles formatting options
   - **Variable resolution:** Resolves variable references in text nodes during output generation
   - **Transformation handling:** Uses special processing for variable references in transformation mode

### State Services

1. **StateService** (`services/state/StateService/`)
   - Stores variables and commands
   - Maintains original and transformed nodes
   - Manages scope and inheritance
   - Tracks file dependencies
   - **Transformation support:** Keeps track of both original and transformed node arrays
   - **Variable copying:** Must explicitly copy variables between parent and child states
   - **Transformation options:** Supports selective transformation of different directive types

2. **StateEventService** (`services/state/StateEventService/`)
   - Handles state change events
   - Manages state updates
   - Provides event hooks
   - Supports state tracking

### Resolution Services

1. **ResolutionService** (`services/resolution/ResolutionService/`)
   - Resolves variables and references
   - Handles path expansions
   - Manages circular dependencies

2. **ValidationService** (`services/resolution/ValidationService/`)
   - Validates directive syntax and constraints
   - Provides extensible validator registration
   - Throws MeldDirectiveError on validation failures
   - Tracks available directive kinds

3. **CircularityService** (`services/resolution/CircularityService/`)
   - Prevents infinite import loops
   - Detects circular variable references
   - Maintains dependency graphs

### File System Services

1. **FileSystemService** (`services/fs/FileSystemService/`)
   - Abstracts file operations (read, write)
   - Supports both real and test filesystems
   - Handles path resolution and validation

2. **PathService** (`services/fs/PathService/`)
   - Validates and normalizes paths
   - Enforces path security constraints
   - Handles path joining and manipulation
   - Supports test mode for path operations

3. **PathOperationsService** (`services/fs/PathOperationsService/`)
   - Handles complex path operations
   - Provides path utilities
   - Manages path transformations

# State Visualization System

## Overview

The State Visualization System provides comprehensive visualization capabilities for Meld's state management system. It enables developers to:

- Visualize state hierarchies and relationships
- Track state transformations over time
- Generate timeline views of state operations
- Calculate and monitor system metrics

## Core Components

### 1. Hierarchy Visualization

Generate hierarchical views of state relationships in multiple formats:

```typescript
const vis = new StateVisualizationService(historyService, trackingService);

// Generate Mermaid diagram
const mermaidDiagram = vis.generateHierarchyView('rootStateId', {
  format: 'mermaid',
  includeMetadata: true,
});

// Generate DOT graph
const dotGraph = vis.generateHierarchyView('rootStateId', {
  format: 'dot',
  includeMetadata: true,
});
```

### 2. Transition Diagrams

Visualize state transformations and their effects:

```typescript
const transitionDiagram = vis.generateTransitionDiagram('stateId', {
  format: 'mermaid',
  includeTimestamps: true,
});
```

### 3. Relationship Graphs

Generate comprehensive views of state relationships:

```typescript
const relationshipGraph = vis.generateRelationshipGraph(['stateId1', 'stateId2'], {
  format: 'mermaid',
  includeMetadata: true,
});
```

### 4. Timeline Views

Create temporal views of state operations:

```typescript
const timeline = vis.generateTimeline(['stateId1', 'stateId2'], {
  format: 'mermaid',
  includeTimestamps: true,
});
```

### 5. Metrics Calculation

Monitor system metrics and state usage:

```typescript
const metrics = vis.getMetrics({
  start: Date.now() - 3600000, // Last hour
  end: Date.now(),
});
```

## Resource Management

### Memory Considerations

The visualization system is designed to be memory-efficient, but there are some important considerations:

1. **History Storage**
   - Operation history is stored in memory
   - Each operation typically consumes ~100-200 bytes
   - Consider clearing old history periodically using `historyService.clearHistoryBefore()`

2. **Visualization Generation**
   - Graph generation is done on-demand
   - Large graphs (>1000 nodes) may require significant memory
   - Consider limiting the scope of visualizations for large state trees

3. **Best Practices**
   - Generate visualizations only when needed
   - Use time ranges to limit data scope
   - Clear old history data periodically
   - Consider using pagination for large datasets

Example of managing history data:

```typescript
// Clear history older than 24 hours
const oneDayAgo = Date.now() - 24 * 60 * 60 * 1000;
historyService.clearHistoryBefore(oneDayAgo);

// Generate visualization with time range
const recentGraph = vis.generateRelationshipGraph(['stateId'], {
  format: 'mermaid',
  timeRange: {
    start: Date.now() - 3600000, // Last hour
    end: Date.now(),
  },
});
```

## Styling and Customization

### Node Styling

Customize node appearance using the `styleNodes` option:

```typescript
const graph = vis.generateRelationshipGraph(['stateId'], {
  format: 'dot',
  styleNodes: (metadata) => ({
    shape: metadata.source === 'merge' ? 'diamond' : 'box',
    color: '#4CAF50',
    tooltip: `Created at: ${new Date(metadata.createdAt).toISOString()}`,
  }),
});
```

### Edge Styling

Customize edge appearance using the `styleEdges` option:

```typescript
const graph = vis.generateRelationshipGraph(['stateId'], {
  format: 'dot',
  styleEdges: (relationship) => ({
    style: relationship.type === 'merge-source' ? 'dashed' : 'solid',
    color: '#2196F3',
    tooltip: `Relationship: ${relationship.type}`,
  }),
});
```

## Output Formats

The system supports multiple output formats:

1. **Mermaid**
   - Markdown-friendly diagram format
   - Great for documentation
   - Easy to embed in GitHub/GitLab

2. **DOT**
   - GraphViz compatible
   - Highly customizable
   - Good for complex graphs

3. **JSON**
   - Machine-readable format
   - Useful for custom rendering
   - Can be transformed to other formats

## Performance Tips

1. **Limit Scope**
   - Use specific state IDs instead of entire trees
   - Apply time ranges to limit data
   - Filter unnecessary metadata

2. **Optimize Generation**
   - Generate visualizations on-demand
   - Cache results when appropriate
   - Use appropriate formats for the use case

3. **Monitor Usage**
   - Track metrics to understand system usage
   - Monitor memory consumption
   - Clear old data regularly

## Error Handling

The system includes robust error handling:

```typescript
try {
  const graph = vis.generateRelationshipGraph(['stateId'], {
    format: 'mermaid',
  });
} catch (error) {
  if (error.message.includes('Unsupported format')) {
    // Handle format error
  } else if (error.message.includes('State not found')) {
    // Handle missing state
  } else {
    // Handle other errors
  }
}
```

## Integration Examples

### 1. Debug View Integration

```typescript
function createDebugView(stateId: string) {
  const hierarchy = vis.generateHierarchyView(stateId, {
    format: 'mermaid',
    includeMetadata: true,
  });

  const timeline = vis.generateTimeline([stateId], {
    format: 'mermaid',
    includeTimestamps: true,
  });

  return `
## State Debug View
### Hierarchy
\`\`\`mermaid
${hierarchy}
\`\`\`

### Timeline
\`\`\`mermaid
${timeline}
\`\`\`
  `;
}
```

### 2. Metrics Dashboard

```typescript
function generateMetricsDashboard() {
  const hourlyMetrics = vis.getMetrics({
    start: Date.now() - 3600000,
    end: Date.now(),
  });

  return {
    totalStates: hourlyMetrics.totalStates,
    statesByType: hourlyMetrics.statesByType,
    avgTransformations: hourlyMetrics.averageTransformationsPerState,
    maxChainLength: hourlyMetrics.maxTransformationChainLength,
    treeDepth: hourlyMetrics.maxTreeDepth,
  };
}
```

# State Visualization System API Documentation

## Overview

The State Visualization System provides a comprehensive API for visualizing and analyzing state operations, relationships, and metrics in the Meld system.

## Core Services

### StateVisualizationService

The main service for generating visualizations and calculating metrics.

```typescript
class StateVisualizationService {
  constructor(
    historyService: StateHistoryService,
    trackingService: StateTrackingService
  );

  // Hierarchy Visualization
  generateHierarchyView(
    rootStateId: string,
    options: {
      format: 'mermaid' | 'dot' | 'json';
      includeMetadata?: boolean;
    }
  ): string;

  // Transition Diagrams
  generateTransitionDiagram(
    stateId: string,
    options: {
      format: 'mermaid' | 'dot' | 'json';
      includeTimestamps?: boolean;
    }
  ): string;

  // Relationship Graphs
  generateRelationshipGraph(
    stateIds: string[],
    options: {
      format: 'mermaid' | 'dot' | 'json';
      includeMetadata?: boolean;
      timeRange?: {
        start: number;
        end: number;
      };
    }
  ): string;

  // Timeline Views
  generateTimeline(
    stateIds: string[],
    options: {
      format: 'mermaid' | 'dot' | 'json';
      includeTimestamps?: boolean;
    }
  ): string;

  // Metrics Calculation
  getMetrics(options: {
    start: number;
    end: number;
  }): StateMetrics;
}
```

### StateMetrics Interface

```typescript
interface StateMetrics {
  totalStates: number;
  statesByType: Map<string, number>;
  averageTransformationsPerState: number;
  maxTransformationChainLength: number;
  averageChildrenPerState: number;
  maxTreeDepth: number;
  operationFrequency: {
    create: number;
    clone: number;
    transform: number;
    merge: number;
  };
}
```

## Output Formats

### Mermaid Format
- Markdown-friendly diagram format
- Great for documentation
- Easy to embed in GitHub/GitLab

Example:
```mermaid
graph TD
  A[State 1] --> B[State 2]
  A --> C[State 3]
  B --> D[State 4]
```

### DOT Format
- GraphViz compatible
- Highly customizable
- Suitable for complex graphs

Example:
```dot
digraph G {
  "State 1" -> "State 2";
  "State 1" -> "State 3";
  "State 2" -> "State 4";
}
```

### JSON Format
- Machine-readable format
- Useful for custom rendering
- Can be transformed to other formats

Example:
```json
{
  "nodes": [
    {"id": "state1", "type": "create"},
    {"id": "state2", "type": "transform"}
  ],
  "edges": [
    {"source": "state1", "target": "state2", "type": "transform"}
  ]
}
```

## Styling Options

### Node Styling
```typescript
interface NodeStyle {
  shape?: 'box' | 'circle' | 'diamond';
  color?: string;
  tooltip?: string;
  label?: string;
}

const graph = vis.generateRelationshipGraph(['stateId'], {
  format: 'dot',
  styleNodes: (metadata) => ({
    shape: metadata.source === 'merge' ? 'diamond' : 'box',
    color: '#4CAF50',
    tooltip: `Created at: ${new Date(metadata.createdAt).toISOString()}`
  })
});
```

### Edge Styling
```typescript
interface EdgeStyle {
  style?: 'solid' | 'dashed' | 'dotted';
  color?: string;
  tooltip?: string;
  label?: string;
}

const graph = vis.generateRelationshipGraph(['stateId'], {
  format: 'dot',
  styleEdges: (relationship) => ({
    style: relationship.type === 'merge-source' ? 'dashed' : 'solid',
    color: '#2196F3',
    tooltip: `Relationship: ${relationship.type}`
  })
});
```

## Error Handling

The service throws specific error types for different scenarios:

```typescript
try {
  const graph = vis.generateRelationshipGraph(['stateId'], {
    format: 'mermaid'
  });
} catch (error) {
  if (error instanceof UnsupportedFormatError) {
    // Handle unsupported format
  } else if (error instanceof StateNotFoundError) {
    // Handle missing state
  } else if (error instanceof VisualizationError) {
    // Handle general visualization errors
  }
}
```

## Best Practices

1. **Memory Management**
   - Generate visualizations on-demand
   - Use time ranges to limit data scope
   - Clear old history data periodically
   - Consider pagination for large datasets

2. **Performance Optimization**
   - Limit visualization scope to specific state IDs
   - Use appropriate formats for the use case
   - Cache results when appropriate
   - Monitor memory consumption

3. **Error Handling**
   - Always wrap visualization calls in try-catch blocks
   - Handle specific error types appropriately
   - Provide meaningful error messages to users
   - Log errors for debugging

4. **Integration Tips**
   - Use Mermaid format for documentation
   - Use DOT format for complex visualizations
   - Use JSON format for custom rendering
   - Consider real-time updates for live systems

\=================================
\=== EXISTING TEST/DEBUG TOOLS ===
Focus:

- Explore StateTrackingService, StateHistoryService, and StateVisualizationService implementations
- Understand current debug capabilities and interfaces

---

# IStateDebuggerService.ts

```typescript
/**
 * @package
 * Interface for state debugging service.
 *
 * @remarks
 * Provides debugging capabilities by integrating state tracking,
 * history, and visualization services. Supports automated diagnostics
 * for failing tests and CLI-based state analysis.
 */

import { VisualizationConfig } from '../StateVisualizationService/IStateVisualizationService';
import { StateOperation } from '../StateHistoryService/IStateHistoryService';
import { StateMetadata } from '../StateTrackingService/IStateTrackingService';

/**
 * Configuration for state capture points
 */
export interface StateCaptureConfig {
  capturePoints: Array<'pre-transform' | 'post-transform' | 'pre-merge' | 'error'>;
  includeFields: Array<'nodes' | 'transformedNodes' | 'variables'>;
  format: 'full' | 'summary';
}

/**
 * Diagnostic result from state analysis
 */
export interface StateDiagnostic {
  stateId: string;
  timestamp: number;
  type: 'error' | 'warning' | 'info';
  message: string;
  context?: {
    operation?: StateOperation;
    metadata?: StateMetadata;
    location?: string;
  };
}

/**
 * Debug session configuration
 */
export interface DebugSessionConfig {
  captureConfig: StateCaptureConfig;
  visualization?: VisualizationConfig;
  traceOperations?: boolean;
  collectMetrics?: boolean;
}

/**
 * Debug session result
 */
export interface DebugSessionResult {
  sessionId: string;
  startTime: number;
  endTime: number;
  diagnostics: StateDiagnostic[];
  snapshots: Map<string, any>;
  metrics?: Record<string, number>;
  visualization?: string;
}

/**
 * Core state debugging service interface
 */
export interface IStateDebuggerService {
  /**
   * Start a new debug session
   * @param config - Debug session configuration
   * @returns Session ID
   */
  startSession(config: DebugSessionConfig): string;

  /**
   * End the current debug session and get results
   * @param sessionId - The session to end
   * @returns Debug session results
   */
  endSession(sessionId: string): Promise<DebugSessionResult>;

  /**
   * Analyze a state for potential issues
   * @param stateId - The state to analyze
   * @returns Array of diagnostics
   */
  analyzeState(stateId: string): Promise<StateDiagnostic[]>;

  /**
   * Trace a state operation and capture debug info
   * @param stateId - The state being operated on
   * @param operation - Function performing the operation
   * @returns Operation result and debug info
   */
  traceOperation<T>(
    stateId: string,
    operation: () => Promise<T>
  ): Promise<{ result: T; diagnostics: StateDiagnostic[] }>;

  /**
   * Get a snapshot of state at a specific point
   * @param stateId - The state to snapshot
   * @param format - Snapshot format ('full' | 'summary')
   * @returns State snapshot
   */
  getStateSnapshot(stateId: string, format: 'full' | 'summary'): Promise<any>;

  /**
   * Generate a CLI-friendly debug report
   * @param sessionId - Debug session to report on
   * @returns Formatted debug report
   */
  generateDebugReport(sessionId: string): Promise<string>;

  /**
   * Register a custom diagnostic analyzer
   * @param analyzer - Function to analyze state and return diagnostics
   */
  registerAnalyzer(
    analyzer: (stateId: string) => Promise<StateDiagnostic[]>
  ): void;

  /**
   * Clear all debug data for a session
   * @param sessionId - Session to clear
   */
  clearSession(sessionId: string): void;
}
```

# StateDebuggerService.test.ts

```typescript
import { describe, it, expect, vi, beforeEach } from 'vitest';
import { StateDebuggerService } from './StateDebuggerService';
import { IStateVisualizationService } from '../StateVisualizationService/IStateVisualizationService';
import { IStateHistoryService } from '../StateHistoryService/IStateHistoryService';
import { IStateTrackingService } from '../StateTrackingService/IStateTrackingService';
import { StateDiagnostic, DebugSessionConfig } from './IStateDebuggerService';

describe('StateDebuggerService', () => {
  // Mock data
  const mockMetadata = {
    type: 'test',
    childStates: ['child1', 'child2'],
    lastModified: Date.now(),
    parentState: null,
    variables: {}
  };

  const mockHistory = {
    transformations: [{ type: 'test', timestamp: Date.now() }],
    operations: []
  };

  // Mock services
  const mockVisualizationService = {
    exportStateGraph: vi.fn().mockReturnValue('graph-data'),
    generateHierarchyView: vi.fn(),
    generateTransitionDiagram: vi.fn(),
    generateRelationshipGraph: vi.fn(),
    generateTimeline: vi.fn(),
    getMetrics: vi.fn(),
  } as unknown as IStateVisualizationService;

  const mockHistoryService = {
    getStateHistory: vi.fn().mockResolvedValue(mockHistory),
  } as unknown as IStateHistoryService;

  const mockTrackingService = {
    getStateMetadata: vi.fn().mockResolvedValue(mockMetadata),
  } as unknown as IStateTrackingService;

  let debugService: StateDebuggerService;
  let testSessionId: string;
  const testStateId = 'test-state-123';

  beforeEach(() => {
    vi.clearAllMocks();
    mockHistoryService.getStateHistory.mockResolvedValue(mockHistory);
    mockTrackingService.getStateMetadata.mockResolvedValue(mockMetadata);
    debugService = new StateDebuggerService(
      mockVisualizationService,
      mockHistoryService,
      mockTrackingService
    );
  });

  describe('Session Management', () => {
    it('should create a new debug session', () => {
      const config: DebugSessionConfig = {
        captureConfig: {
          capturePoints: ['pre-transform', 'post-transform'],
          includeFields: ['nodes'],
          format: 'full'
        }
      };

      const sessionId = debugService.startSession(config);
      expect(sessionId).toBeDefined();
      expect(typeof sessionId).toBe('string');
    });

    it('should end a debug session and return results', async () => {
      const config: DebugSessionConfig = {
        captureConfig: {
          capturePoints: ['pre-transform'],
          includeFields: ['nodes'],
          format: 'full'
        },
        visualization: { format: 'mermaid' }
      };

      const sessionId = debugService.startSession(config);
      const result = await debugService.endSession(sessionId);

      expect(result).toMatchObject({
        sessionId,
        startTime: expect.any(Number),
        endTime: expect.any(Number),
        diagnostics: expect.any(Array),
        snapshots: expect.any(Map),
      });

      expect(mockVisualizationService.exportStateGraph).toHaveBeenCalledWith(
        config.visualization
      );
    });

    it('should throw error when ending non-existent session', async () => {
      await expect(debugService.endSession('invalid-id'))
        .rejects
        .toThrow('No debug session found');
    });
  });

  describe('State Analysis', () => {
    it('should analyze state and return diagnostics', async () => {
      // Setup mock to trigger warnings
      mockHistoryService.getStateHistory.mockResolvedValueOnce({
        transformations: Array(11).fill({ type: 'test' }),
        operations: []
      });
      mockTrackingService.getStateMetadata.mockResolvedValueOnce({
        type: 'test',
        childStates: Array(21).fill('child'),
        lastModified: Date.now(),
        parentState: null,
        variables: {}
      });

      const diagnostics = await debugService.analyzeState(testStateId);

      expect(diagnostics).toHaveLength(2); // Two warnings
      expect(diagnostics.at(0).type).toBe('warning');
      expect(diagnostics.at(0).message).toContain('transformations');
      expect(diagnostics.at(1).type).toBe('warning');
      expect(diagnostics.at(1).message).toContain('child states');
    });

    it('should run custom analyzers during analysis', async () => {
      const customAnalyzer = vi.fn().mockResolvedValue([{
        stateId: testStateId,
        timestamp: Date.now(),
        type: 'info',
        message: 'Custom analysis'
      }]);

      debugService.registerAnalyzer(customAnalyzer);
      const diagnostics = await debugService.analyzeState(testStateId);

      expect(customAnalyzer).toHaveBeenCalledWith(testStateId);
      expect(diagnostics).toContainEqual(expect.objectContaining({
        message: 'Custom analysis'
      }));
    });

    it('should handle missing state data', async () => {
      mockHistoryService.getStateHistory.mockResolvedValueOnce(null);
      mockTrackingService.getStateMetadata.mockResolvedValueOnce(null);

      const diagnostics = await debugService.analyzeState(testStateId);

      expect(diagnostics).toHaveLength(1);
      expect(diagnostics.at(0).type).toBe('error');
      expect(diagnostics.at(0).message).toContain('Failed to retrieve state');
    });
  });

  describe('Operation Tracing', () => {
    it('should trace successful operations', async () => {
      const operation = vi.fn().mockResolvedValue('success');

      const { result, diagnostics } = await debugService.traceOperation(
        testStateId,
        operation
      );

      expect(result).toBe('success');
      expect(diagnostics).toEqual(expect.any(Array));
      expect(mockTrackingService.getStateMetadata).toHaveBeenCalled();
      expect(mockHistoryService.getStateHistory).toHaveBeenCalled();
    });

    it('should handle failed operations', async () => {
      const error = new Error('Operation failed');
      const operation = vi.fn().mockRejectedValue(error);

      try {
        await debugService.traceOperation(testStateId, operation);
        fail('Expected operation to throw');
      } catch (e: any) {
        expect(e).toMatchObject({
          error,
          diagnostics: expect.arrayContaining([
            expect.objectContaining({
              type: 'error',
              message: 'Operation failed'
            })
          ])
        });
      }
    });

    it('should handle missing state data during tracing', async () => {
      mockHistoryService.getStateHistory.mockResolvedValue(null);
      mockTrackingService.getStateMetadata.mockResolvedValue(null);
      const operation = vi.fn().mockResolvedValue('success');

      try {
        await debugService.traceOperation(testStateId, operation);
        fail('Expected operation to throw');
      } catch (e: any) {
        expect(e.error.message).toContain('Failed to retrieve state data');
      }
    });
  });

  describe('State Snapshots', () => {
    it('should get full state snapshot', async () => {
      const snapshot = await debugService.getStateSnapshot(testStateId, 'full');

      expect(snapshot).toMatchObject({
        metadata: mockMetadata,
        history: mockHistory,
        children: expect.any(Array)
      });

      expect(snapshot.children).toHaveLength(mockMetadata.childStates.length);
    });

    it('should get summary state snapshot', async () => {
      const snapshot = await debugService.getStateSnapshot(testStateId, 'summary');

      expect(snapshot).toMatchObject({
        id: testStateId,
        type: mockMetadata.type,
        childCount: mockMetadata.childStates.length,
        transformationCount: mockHistory.transformations.length,
        lastModified: expect.any(Number)
      });
    });

    it('should handle missing state data in snapshots', async () => {
      mockHistoryService.getStateHistory.mockResolvedValue(null);
      mockTrackingService.getStateMetadata.mockResolvedValue(null);

      await expect(debugService.getStateSnapshot(testStateId, 'full'))
        .rejects
        .toThrow('Failed to retrieve state data');
    });
  });

  describe('Debug Reports', () => {
    it('should generate debug report', async () => {
      const config: DebugSessionConfig = {
        captureConfig: {
          capturePoints: ['pre-transform'],
          includeFields: ['nodes'],
          format: 'full'
        }
      };

      const sessionId = debugService.startSession(config);
      const report = await debugService.generateDebugReport(sessionId);

      expect(report).toContain('Debug Session Report');