meld/_dev/notes/dev/old/INSTR.md

Meld: A template language for LLM prompts

# State Instrumentation Plan 

## Overview & Motivation [✅ Complete]

The goal of this instrumentation layer is to provide deep visibility into state transitions, transformations, and lifecycle events within Meld's state management system. This will help us:

- Debug state-related issues more effectively
- Understand complex state transitions
- Verify transformation behavior
- Support long-term maintenance
- Enable data-driven improvements

## Core Principles [✅ Complete]

1. **Incremental Implementation** [✅ Complete]
   - Start with critical operations (clone, transform)
   - Add capabilities progressively 
   - Maintain existing test stability
   - Allow gradual adoption 

2. **Backward Compatibility** [✅ Complete]
   - Support existing test patterns 
   - Clear migration paths 
   - Explicit removal targets 
   - Controlled deprecation process

3. **Testing First** [✅ Complete]
   - Instrumentation must be thoroughly tested 
   - No degradation of existing test coverage 
   - Support for both old and new patterns during migration 

4. **Documentation & Clarity** [✅ Complete]
   - Clear JSDoc for all new components 
   - Migration annotations for legacy code 
   - Explicit compatibility notes 
   - Usage examples and patterns 

## Guidelines [✅ Complete]

### Documentation Standards [✅ Complete]

1. New Components: 
```typescript
/**
 * @package
 * Tracks state lifecycle events and transitions.
 * 
 * @remarks
 * Core component of the state instrumentation system. Provides
 * event emission, history tracking, and debugging capabilities.
 * 
 * @example
 * ```ts
 * const tracker = new StateLifecycleTracker();
 * tracker.onTransformation(event => {
 *   console.log(`Node transformed: ${event.context}`);
 * });
 * ```
 */
```

2. Migration-Targeted Code: 
```typescript
/**
 * @deprecated
 * Legacy state tracking implementation.
 * Will be removed once all tests use StateLifecycleTracker.
 * 
 * @see {@link StateLifecycleTracker} for the new implementation
 * @removal-target Phase 3 - Test Infrastructure Migration
 */
```

### Backward Compatibility [✅ Complete]

1. **Identification** [✅ Complete]
   - Mark legacy code with `@deprecated` 
   - Document replacement patterns 
   - Specify removal phase 
   - Track usage patterns 

2. **Migration Support** [✅ Complete]
   - Provide migration utilities 
   - Support running in both modes
   - Clear upgrade paths 
   - Validation tools 

3. **Removal Process** [✅ Complete]
   - Phase-specific removal targets 
   - Usage monitoring 
   - Explicit dependencies 
   - Clean migration paths 

### Testing Requirements [✅ Complete]

1. **Coverage Requirements** [✅ Complete]
   - Full coverage of new components 
   - Migration utility testing 
   - Integration validation 

2. **Test Infrastructure** [✅ Complete]
   - Support both old and new patterns 
   - Clear test utilities
   - Migration helpers 

## Implementation Phases 

Each phase of implementation follows these structural elements:

1. **Purpose & Objectives**
   - Clear goals
   - Success metrics
   - Risk assessment
   - Value proposition

2. **Requirements**
   - Functional requirements
   - Non-functional requirements
   - Migration requirements
   - Testing requirements

3. **Exit Criteria**
   - Specific deliverables
   - Quality metrics
   - Migration progress

4. **Testing Strategy**
   - Coverage requirements
   - Migration validation
   - Integration testing

5. **Migration Considerations**
   - Backward compatibility
   - Upgrade paths
   - Removal targets
   - Validation approach 

## Phase 1: Core Event Infrastructure [✅ Complete]

### Purpose & Objectives [✅ Complete]

Build the foundational event system to track state transitions and transformations. This phase focuses on:
- Event emission [✅ Complete]
- Event handling [✅ Complete]
- Basic instrumentation [✅ Complete]
- Core event types [✅ Complete]

### Core Components [✅ Complete]

A) Event System [✅ Complete]
   • Event emission [✅ Complete]
   • Event handling [✅ Complete]
   • Event types:
     ```typescript
     type StateEvent = {
       type: 'create' | 'clone' | 'transform';
       stateId: string;
       source: string;
       timestamp: number;
     };
     ```

B) Event Handling [✅ Complete]
   • Event registration [✅ Complete]
   • Event dispatch [✅ Complete]
   • Basic filtering [✅ Complete]
   • Handler management:
     ```typescript
     interface EventHandler {
       onEvent(event: StateEvent): void;
       filter?: (event: StateEvent) => boolean;
     }
     ```

C) Basic Instrumentation [✅ Complete]
   • Event logging [✅ Complete]
   • Basic metrics [✅ Complete]
   • Error tracking [✅ Complete]
   • Debug output [✅ Complete]

### Requirements [✅ Complete]

1. Functional Requirements [✅ Complete]
   - Event emission
   - Event handling
   - Basic logging
   - Error tracking

2. Non-Functional Requirements [✅ Complete]
   - Clear event flow
   - Easy registration
   - Simple debugging
   - Error handling
   - Memory management [✅ Complete]

3. Migration Requirements [✅ Complete]
   - Support existing code
   - Clear upgrade path
   - No breaking changes
   - Documentation

4. Testing Requirements [✅ Complete]
   - Event validation
   - Handler testing
   - Error handling
   - Integration tests

### Exit Criteria [✅ Complete]

1. Implementation [✅ Complete]
   - Events working
   - Handlers working
   - Logging working
   - Integration done

2. Testing [✅ Complete]
   - All tests passing
   - Events verified
   - Handlers tested
   - Integration verified

3. Documentation [✅ Complete]
   - Events documented
   - Handlers guide
   - Debug guide
   - Migration guide

4. Validation [✅ Complete]
   - Events verified
   - Handlers working
   - Logging verified
   - Integration complete

### Testing Strategy

1. Event Testing
   - Event emission
   - Event handling
   - Event filtering
   - Basic metrics

2. Integration Testing
   - Handler integration
   - Service integration
   - Tool integration
   - Migration validation

3. Error Testing
   - Error handling
   - Error logging
   - Debug output
   - Recovery paths

### Example Implementation Patterns

1. Event System:
```typescript
/**
 * @package
 * Core event system for state tracking.
 * 
 * @remarks
 * Provides event emission and handling for state operations.
 * 
 * @example
 * ```ts
 * const events = new EventSystem();
 * events.on('transform', event => {
 *   console.log(`State ${event.stateId} transformed`);
 * });
 * ```
 */
```

2. Event Handling:
```typescript
/**
 * @package
 * Event handler registration and management.
 * 
 * @remarks
 * Manages event handlers and filtering.
 * 
 * @example
 * ```ts
 * const handler = new EventHandler();
 * handler.register({
 *   onEvent: event => console.log(event),
 *   filter: event => event.type === 'transform'
 * });
 * ```
 */
```

### Migration Notes

1. Current Events
```typescript
/**
 * @deprecated
 * Legacy event handling implementation.
 * Will be replaced by EventSystem in Phase 2.
 * 
 * @see {@link EventSystem}
 * @removal-target Phase 2
 */
```

2. Transition Strategy
- Add events alongside existing code
- Support both approaches
- Validate all scenarios
- Remove old code after migration

## Phase 2: State Tracking Enhancement [✅ Complete]

### Purpose & Objectives

Build on the event system to track state relationships and transitions. This phase focuses on:
- State instance tracking [✅ Complete]
- Parent-child relationships [✅ Complete]
- State lineage tracking [✅ Complete]
- State transition history [✅ Complete]
- Relationship visualization [✅ Complete]

### Core Components 

A) State Instance Tracking [✅ Complete]
   • Unique state identification [✅ Complete]
   • Relationship tracking [✅ Complete]
   • State lineage tracking [✅ Complete]
   • Basic state info [✅ Complete]

B) State History [✅ Complete]
   • Operation history tracking [✅ Complete]
   • State relationships [✅ Complete]
   • Transition records [✅ Complete]
   • History structure [✅ Complete]
     
C) Visualization Support [✅ Complete]
   • State hierarchy views [✅ Complete]
   • Transition diagrams [✅ Complete]
   • Relationship graphs [✅ Complete]
   • Basic metrics [✅ Complete]

### Requirements 

1. Functional Requirements [✅ Complete]
   - Unique state identification [✅ Complete]
   - Relationship tracking [✅ Complete]
   - Lineage tracking [✅ Complete]
   - History recording [✅ Complete]
   - Visualization support [✅ Complete]

2. Non-Functional Requirements [✅ Complete]
   - Error handling [✅ Complete]
   - Debug support [✅ Complete]
   - Memory management [✅ Complete]

3. Migration Requirements [✅ Complete]
   - Legacy code support [✅ Complete]
   - Transition utilities [✅ Complete]
   - Documentation updates [✅ Complete]
   - Test adaptation [✅ Complete]

4. Testing Requirements [✅ Complete]
   - Unit test coverage [✅ Complete]
   - Integration tests [✅ Complete]
   - Migration tests [✅ Complete]

### Next Steps

1. Documentation [✅ Complete]
   - Complete API documentation [✅ Complete]
   - Finish migration guides [✅ Complete]
   - Add usage examples [✅ Complete]

### Exit Criteria [✅ Complete]

1. Implementation
   - Core tracking complete
   - History recording working
   - Visualization tools ready
   - Migration utilities done

2. Testing
   - All tests passing
   - Migration validated
   - Integration confirmed

3. Documentation
   - API documentation
   - Migration guides
   - Debug guides
   - Example code

4. Validation
   - Error rates
   - Migration success
   - User feedback

### Testing Strategy 

1. Tracking Testing [✅ Complete]
   - ID generation
   - Relationship tracking
   - History recording
   - Basic metrics

2. Integration Testing [✅ Complete]
   - Event system integration
   - Service integration
   - Tool integration
   - Migration validation 

3. Visualization Testing [✅ Complete]
   - Graph generation
   - Relationship display
   - History views
   - Basic metrics

### Example Implementation Patterns

1. State Tracking:
```typescript
/**
 * @package
 * Tracks state instances and relationships.
 * 
 * @remarks
 * Provides unique identification and relationship tracking
 * for state instances.
 * 
 * @example
 * ```ts
 * const tracker = new StateTracker();
 * tracker.onNewState(state => {
 *   console.log(`New state: ${state.id}`);
 *   if (state.parentId) {
 *     console.log(`Parent: ${state.parentId}`);
 *   }
 * });
 * ```
 */
```

2. History Recording:
```typescript
/**
 * @package
 * Records state operation history.
 * 
 * @remarks
 * Tracks operations and relationships between states.
 * 
 * @example
 * ```ts
 * const history = new StateHistory();
 * history.recordOperation({
 *   type: 'clone',
 *   source: 'directive',
 *   parentId: originalState.id
 * });
 * ```
 */
```

### Migration Notes

1. Current Tracking
```typescript
/**
 * @deprecated
 * Legacy state tracking implementation.
 * Will be replaced by StateTracker in Phase 3.
 * 
 * @see {@link StateTracker}
 * @removal-target Phase 3
 */
```

2. Transition Strategy
- Add tracking alongside existing code
- Support both approaches
- Validate all scenarios
- Remove old code after migration

## Phase 3: Debugging Infrastructure & Visualization [✅ Complete]

### Purpose & Objectives

Build debugging tools that leverage the event system and state tracking to provide clear visibility into state transitions and transformations. This phase focuses on:
- Debugging tools for state transitions
- Visual representation of state hierarchies
- Automated diagnostic tools for failing tests
- Clear state visualization

### Core Components

A) Debug Tooling
   • CLI tools for analyzing state history
   • Visual representation of state transitions
   • Integration with existing debug logging
   • State operation tracking:
     ```typescript
     {
       operationType: 'clone' | 'transform' | 'merge';
       source: string;
       context: string;
       location?: SourceLocation;
       stateId: string;
       parentStateId?: string;
     }
     ```

B) Snapshot System
   • State snapshots at key points
   • Comparison utilities for state diffs
   • Integration with existing MemfsTestFileSystem
   • Capture configuration:
     ```typescript
     {
       capturePoints: ['pre-transform', 'post-transform', 'pre-merge', 'error'];
       includeFields: ['nodes', 'transformedNodes', 'variables'];
       format: 'full' | 'summary';
     }
     ```

C) Diagnostic Tools
   • Automatic state history for failing tests
   • Root cause analysis helpers
   • State transition visualizers
   • State relationship diagrams

### Requirements

1. Functional Requirements
   - Clear CLI debugging interface
   - Visual state hierarchy tools
   - Automated diagnostic capabilities
   - State transition tracking

2. Non-Functional Requirements
   - Clear visualization output
   - Helpful error messages
   - Intuitive debug workflows
   - Easy test integration

3. Migration Requirements
   - Support existing debugging patterns
   - Integration with current tools
   - Clear upgrade path
   - Documentation of new workflows

4. Testing Requirements
   - Tool reliability validation
   - Integration test coverage
   - Error handling verification
   - Debug workflow validation

### Exit Criteria

1. Implementation
   - Debug tools functional
   - Visualization system working
   - Diagnostic capabilities tested
   - State tracking validated

2. Testing
   - All tools thoroughly tested
   - Integration tests passing
   - Error handling verified
   - Debug workflows validated

3. Documentation
   - Complete debugging guide
   - Tool usage documentation
   - Debug workflow guides
   - Migration workflows documented

4. Validation
   - Tool effectiveness verified
   - Debug workflows validated
   - State tracking verified
   - Documentation complete

### Testing Strategy

1. Tool Testing
   - CLI tool validation
   - Visualization accuracy
   - Diagnostic tool reliability
   - State tracking verification

2. Integration Testing
   - Tool chain integration
   - Debug workflow validation
   - Error handling verification
   - State tracking validation

3. User Workflow Testing
   - Debug scenario coverage
   - Tool chain effectiveness
   - Migration path validation
   - Error handling verification

### Example Implementation Patterns

1. Debug Tool Configuration:
```typescript
/**
 * @package
 * Configures state debugging tools and captures.
 * 
 * @remarks
 * Controls what information is captured and how it's presented.
 * Focuses on making state transitions clear and debuggable.
 * 
 * @example
 * ```ts
 * const debugConfig = {
 *   capturePoints: ['pre-transform', 'post-transform'],
 *   visualization: {
 *     showLineage: true,
 *     highlightChanges: true
 *   }
 * };
 * ```
 */
```

2. Diagnostic Integration:
```typescript
/**
 * @package
 * Integrates diagnostic tools with test infrastructure.
 * 
 * @remarks
 * Provides automatic diagnostic information for failing tests.
 * Helps identify root causes of state-related failures.
 * 
 * @example
 * ```ts
 * test('should handle complex transformation', async () => {
 *   const diagnostics = await StateDebugger.trace(async () => {
 *     const state = new StateService();
 *     await complexOperation(state);
 *   });
 *   
 *   // Automatic analysis of state transitions
 *   expect(diagnostics).not.toHaveStateInconsistencies();
 *   expect(diagnostics).toShowExpectedTransformations();
 * });
 * ```
 */
```

### Migration Notes

1. Current Debug Tools
```typescript
/**
 * @deprecated
 * Legacy debugging implementation.
 * Will be replaced by StateDebugger in Phase 4.
 * 
 * @see {@link StateDebugger}
 * @removal-target Phase 4
 */
```

2. Transition Strategy
- Introduce new tools alongside existing ones
- Support both debugging approaches
- Validate all debug scenarios
- Remove old tools after full migration

## Phase 4: Production Readiness & Stabilization [🚧 In Progress]

### Purpose & Objectives

Finalize the instrumentation system for production use, complete migrations, and add token tracking for AI model compatibility. This phase focuses on:
- Token tracking and model compatibility
- Final legacy code migration
- Complete documentation
- Stability verification

### Core Components

A) Token Tracking
   • Character and token counting
   • Model compatibility checking
   • Clear limit indicators
   • Simple status output:
     ```typescript
     {
       characters: number;
       tokens: number;
       modelCompatibility: {
         claude: boolean;    // ✔/❌ for 200k tokens
         gpt4: boolean;      // ✔/❌ for 200k tokens
         palm: boolean;      // ✔/❌ for 1M tokens
       }
     }
     ```

B) Model Configuration
   • Simple model limit definitions
   • Easy limit updates
   • Clear status reporting:
     ```typescript
     {
       modelLimits: {
         claude: 200000,
         gpt4: 200000,
         palm: 1000000
       }
     }
     ```

C) Migration Completion
   • Legacy code removal
   • Pattern standardization
   • Test modernization
   • Verification tools

### Requirements

1. Functional Requirements
   - Accurate token counting
   - Simple model compatibility checks
   - Clear status reporting
   - Migration verification utilities

2. Non-Functional Requirements
   - Clear status messages
   - Easy model limit updates
   - Robust error handling
   - Simple configuration

3. Migration Requirements
   - Complete legacy code removal
   - All tests using new patterns
   - No deprecated APIs in use
   - Clean architecture

4. Documentation Requirements
   - Complete API documentation
   - Model limit documentation
   - Token tracking guide
   - Migration completion report

### Exit Criteria

1. Implementation
   - Token tracking working
   - Model checks functional
   - Migration complete
   - Documentation thorough

2. Testing
   - All tests passing
   - Token counting verified
   - Model checks tested
   - No legacy code in use

3. Documentation
   - Full API documentation
   - Token tracking guides
   - Model compatibility documented
   - Migration guides complete

4. Validation
   - Token counting accuracy verified
   - Model checks validated
   - Migration completed
   - Documentation verified

### Testing Strategy

1. Token Analysis Testing
   - Character counting accuracy
   - Token counting accuracy
   - Model limit checking
   - Status reporting

2. Migration Testing
   - Legacy code scan
   - Pattern verification
   - API usage validation
   - Integration verification

3. System Testing
   - Large document handling
   - Status reporting clarity
   - Error handling verification
   - Integration validation

### Example Implementation Patterns

1. Token Analysis:
```typescript
/**
 * @package
 * Tracks token usage and model compatibility.
 * 
 * @remarks
 * Simple token counting and model limit checking,
 * similar to the cpai tool's output format.
 * 
 * @example
 * ```ts
 * const analysis = await TokenAnalyzer.analyze(state);
 * console.log(`📋 ${analysis.characters} characters ` +
 *             `(${analysis.tokens} tokens)`);
 * console.log('Input limits:',
 *   `${analysis.modelCompatibility.claude ? '✔' : '❌'} Claude`,
 *   `${analysis.modelCompatibility.gpt4 ? '✔' : '❌'} GPT-4`);
 * ```
 */
```

2. Migration Verification:
```typescript
/**
 * @package
 * Verifies completion of instrumentation migration.
 * 
 * @remarks
 * Ensures all legacy patterns have been replaced and
 * validates the new implementation is complete.
 * 
 * @example
 * ```ts
 * const verification = await MigrationVerifier.scan({
 *   checkDeprecated: true,
 *   validatePatterns: true,
 *   requireModernAPI: true
 * });
 * 
 * expect(verification).toBeFullyMigrated();
 * ```
 */
```

### Final Migration Notes

1. Verification Process
- Scan for deprecated APIs
- Validate all new patterns
- Verify token tracking
- Confirm status reporting

2. Cleanup Process
- Remove deprecated code
- Clean up migration utilities
- Archive migration docs
- Update API documentation

### Production Considerations

1. Token Tracking
   - Accurate counting
   - Clear status reporting
   - Easy model updates
   - Simple configuration

2. Stability
   - Error handling
   - Status clarity
   - Update handling
   - Documentation maintenance

3. Maintenance
   - Model limit updates
   - Status format updates
   - Simple configuration
   - Documentation updates

### Long-term Maintenance

1. Model Compatibility
   - Track token limit changes
   - Update model configurations
   - Maintain status format
   - Update documentation

2. Update Process
   - Version compatibility
   - Migration tools
   - Testing procedures
   - Documentation updates

3. Support Requirements
   - Token tracking tools
   - Status reporting
   - Model updates
   - Documentation maintenance