meld/_dev/notes/dev/old/instrumentation.md

Meld: A template language for LLM prompts

─────────────────────────────────────────────────────────────────────────
2) RECOMMENDED LONG-TERM STATE INSTRUMENTATION APPROACH
─────────────────────────────────────────────────────────────────────────

As you move into the next phases, you have an opportunity to build a small “state instrumentation” layer. This would allow you to:

• Trace every significant change to the state (e.g., transformations, variable updates, clone operations) in a structured, consistent manner.  
• Easily debug tests by dumping or logging the exact shape of the state at key lifecycle points (e.g., after parse, after interpret, after transform).  
• Provide snapshot-based comparisons to detect unintended changes or partial transformations.

Below is a detailed outline for implementing such instrumentation:

A) Add a “StateLifecycleEvent” Emitter  
   • Within StateService, introduce a small event mechanism (e.g., an EventEmitter or an observer API).  
   • Whenever “createChildState,” “mergeChildState,” “enableTransformation,” or “transformNode” is called, emit a structured event:  
     {
       eventType: 'enableTransformation' | 'transformNode' | 'clone' | ...
       oldStateSnapshot: Partial<StateNode> | undefined
       newStateSnapshot: Partial<StateNode> | undefined
       context: string (e.g., reason or directive info)
     }  
   • If you want minimal overhead, you can store just “state deltas” instead of full snapshots.

B) Attach a Logger or Debug Writer  
   • Provide an injectable or configurable logger (e.g., a “StateInstrumentationService” or “StateLogger”) that listens for these events.  
   • By default, it might log to console.debug in dev/test, or remain silent in production (unless debug logging is enabled).  
   • For persistent debugging, you can write these events into a .json or .log file.

C) Support Unique Trace IDs for Each StateService Instance  
   • When you call createChildState() or clone(), mark the resulting new instance with a “serviceId” or an incrementing “childNumber.”  
   • This helps you differentiate logs from parent and child states. This can be crucial when debugging multiple nested imports or merges.

D) Expose Instrumentation Hooks in Tests
   • For integration tests, you might redirect instrumentation logs to a buffer or an in-memory store.  
   • On a test failure, print the last few events (the “history” of state changes) to see exactly where transformations or merges went wrong.

E) Summarize and Compare Snapshots  
   • You can integrate with your existing MemfsTestFileSystem or TestSnapshot system to store state snapshots (like an in-memory “debugState.json”).  
   • If a transformation unexpectedly modifies a variable or fails to replace a directive, you’ll see a difference in the final snapshot.  

F) Roll This Out Incrementally  
   • Start by instrumenting the clone() and transformNode() calls, since these are the biggest sources of confusion.  
   • Next, add events for key merges (mergeChildState) or transformations (enableTransformation).  
   • Over time, you can unify that instrumentation into a single “onStateChange” observer.

This approach, if done carefully, will significantly reduce detective work each time you see a test fail with “expected 'test output' but got 'echo test'.” You’ll be able to see the entire chain of transformations or to confirm that no transformNode call was triggered for that directive.


==== REWRITTEN PLAN ====

# State Instrumentation Implementation Plan

## Overview
This plan details the implementation of a state instrumentation layer to improve debugging, testing, and maintenance of Meld's state management. The goal is to make state transitions traceable, testable, and debuggable while maintaining our existing test coverage and performance.

## Phase 1: Core Event Infrastructure

### Requirements
1. StateLifecycleEvent Interface
   - Define precise event types for all state operations
   - Include source location, operation context, and timing
   - Support both full snapshots and deltas
   - Maintain type safety throughout the event system

2. Event Emitter Integration
   - Minimal impact on StateService's public interface
   - Thread-safe event emission
   - Support for synchronous and asynchronous handlers
   - No performance impact when no listeners are attached

3. Base Instrumentation Service
   - Injectable logger/instrumentation service
   - Configuration system for enabling/disabling features
   - Memory-efficient event buffering
   - Support for different output formats (JSON, structured logs)

### Success Criteria
- Complete test coverage of event infrastructure
- Zero impact on existing 504 passing tests
- Type-safe event handling verified by TypeScript
- Performance benchmarks showing negligible overhead
- Unit tests for all new components

### Testing Requirements
1. Event Emission Tests
   - Verify correct event types for each operation
   - Test event payload accuracy
   - Validate timing and ordering
   - Check memory usage patterns

2. Integration Tests
   - Verify StateService integration
   - Test configuration system
   - Validate logging output
   - Check performance impact

## Phase 2: State Tracking Enhancement

### Requirements
1. Unique State Instance Tracking
   - Generate and maintain unique IDs for state instances
   - Track parent-child relationships
   - Support state clone lineage tracking
   - Maintain state hierarchy information

2. State Delta Tracking
   - Efficient storage of state changes
   - Support for rollback/history
   - Clear differentiation between original and transformed states
   - Tracking of transformation enablement

3. Mock Alignment
   - Update mock StateService to support instrumentation
   - Ensure consistent behavior between real and mock services
   - Add instrumentation interfaces to test utilities
   - Maintain backward compatibility

### Success Criteria
- Complete tracking of state instance relationships
- Accurate state history in test scenarios
- Mock services fully aligned with real implementations
- No regression in existing tests
- Performance within acceptable bounds

### Testing Requirements
1. State Tracking Tests
   - Verify unique ID generation
   - Test parent-child relationships
   - Validate clone tracking
   - Check state hierarchy maintenance

2. Mock Service Tests
   - Verify instrumentation support in mocks
   - Test consistency with real services
   - Validate backward compatibility
   - Check test utility integration

## Phase 3: Debugging Infrastructure

### Requirements
1. Test Integration
   - Automatic capture of state history in failing tests
   - Integration with existing test snapshot system
   - Support for custom assertion helpers
   - Debug output formatting

2. Snapshot System
   - Efficient storage of state snapshots
   - Comparison utilities for state diffs
   - Integration with existing MemfsTestFileSystem
   - Support for selective snapshot capture

3. Debug Tooling
   - CLI tools for analyzing state history
   - Visual representation of state transitions
   - Integration with existing debug logging
   - Performance profiling support

### Success Criteria
- Improved debugging experience for state-related failures
- Automatic state history in test failures
- Efficient snapshot storage and comparison
- Minimal impact on test execution time

### Testing Requirements
1. Debug Integration Tests
   - Verify automatic capture in failures
   - Test snapshot system integration
   - Validate debug output format
   - Check performance impact

2. Tool Integration Tests
   - Test CLI tool functionality
   - Verify visualization accuracy
   - Validate profiling support
   - Check integration with existing tools

## Phase 4: Production Integration

### Requirements
1. Production Configuration
   - Production-safe default settings
   - Performance optimization options
   - Configurable logging levels
   - Resource usage limits

2. Error Integration
   - Enhanced error messages with state context
   - Automatic inclusion of relevant state history
   - Integration with existing error handling
   - Support for error recovery

3. Documentation
   - Clear usage guidelines
   - Performance impact documentation
   - Debug workflow documentation
   - Migration guides for existing code

### Success Criteria
- Production-ready configuration
- Minimal performance impact in production
- Clear documentation and examples
- Smooth migration path for existing code

### Testing Requirements
1. Production Tests
   - Verify performance in production mode
   - Test resource usage limits
   - Validate error integration
   - Check configuration system

2. Documentation Tests
   - Verify example code
   - Test migration procedures
   - Validate debug workflows
   - Check API documentation

## Implementation Notes

### Backward Compatibility
- All changes must maintain compatibility with existing tests
- Gradual adoption path for new features
- Support for legacy test patterns
- Clear migration documentation

### Performance Considerations
- Minimal overhead when disabled
- Efficient event storage and processing
- Smart memory management
- Configurable resource limits

### Integration Points
- Test framework integration
- Existing logging system integration
- Error handling system integration
- Development tools integration

### Quality Assurance
- Comprehensive test coverage
- Performance benchmarking
- Memory usage monitoring
- Error handling verification

==== ADDITION ====

# State Instrumentation Implementation Plan - Migration Strategy

## Documentation & Migration Philosophy

### Core Principles
1. Every new component must be fully documented with its intended final form
2. All temporary compatibility code must be clearly marked and documented
3. Migration paths should be explicit and testable
4. Changes should be removable in clear, discrete steps

### Documentation Requirements

1. New Component Documentation
```typescript
/**
 * @package
 * 
 * State lifecycle event emitter for tracking state transitions.
 * 
 * @remarks
 * This is part of the new state instrumentation system. It replaces the previous
 * ad-hoc debug logging in StateService. Once all tests are migrated, this will
 * become the sole source of state transition information.
 * 
 * @example
 * ```ts
 * const emitter = new StateLifecycleEmitter();
 * emitter.on('stateCloned', (event) => {
 *   // Handle clone event
 * });
 * ```
 */
```

2. Compatibility Layer Documentation
```typescript
/**
 * @deprecated
 * Temporary compatibility method to support existing tests.
 * Will be removed once all tests are migrated to use StateLifecycleEmitter.
 * 
 * @see {@link StateLifecycleEmitter} for the new implementation
 * @removal-target Phase 3 - Test Infrastructure Migration
 */
```

3. Migration Tags
```typescript
// @MIGRATION-TARGET: Phase 2
// This mock implementation maintains compatibility with existing tests
// by implementing both old and new interfaces. The old interface can be
// removed once all tests in services/**/*.test.ts are migrated to use
// the new StateLifecycleEmitter.
```

### Phase-Specific Migration Documentation

Each phase should include:

1. Migration Manifest
```markdown
## Phase 2 Migration Targets

### Components to Remove
- `StateService._debugLog` method
- `TestContext.mockStateDebug` utility
- Legacy state snapshot format

### Required Changes
- Update all directive handler tests to use new event system
- Migrate existing debug logs to event listeners
- Convert snapshot assertions to new format

### Migration Verification
- No usages of deprecated APIs in new tests
- All specified components marked with removal tags
- Migration tests pass with both old and new systems
```

2. Verification Checklist
```markdown
### Migration Verification Steps

1. Component Isolation
   - [ ] New component has no dependencies on legacy system
   - [ ] Legacy system has clear boundaries with new system
   - [ ] All touch points documented with @removal-target tags

2. Test Coverage
   - [ ] Tests exist for both old and new paths
   - [ ] Migration tests verify both systems work
   - [ ] No new usage of deprecated features

3. Documentation
   - [ ] All new APIs fully documented
   - [ ] Migration paths documented
   - [ ] Removal targets identified
```

### Implementation Guidelines

1. Code Organization
```typescript
// New implementation
src/
  state/
    instrumentation/
      // All new code here, clean implementation
    legacy/
      // Compatibility layer, clearly marked for removal
    migrations/
      // Migration utilities and verification tests
```

2. Test Organization
```typescript
tests/
  state/
    instrumentation/
      // Pure new system tests
    migrations/
      // Tests that verify both systems
    legacy/
      // Tests that must be migrated, grouped by phase
```

3. Migration Utilities
```typescript
/**
 * @internal
 * Utility to help migrate tests from old to new state tracking.
 * 
 * @remarks
 * This helper exists solely to make migration easier and more reliable.
 * It will be removed along with the legacy system in Phase 4.
 * 
 * @example
 * ```ts
 * const migrationHelper = new StateMigrationHelper();
 * await migrationHelper.migrateTest({
 *   oldTest: existingTest,
 *   newSystem: true,
 *   verifyBoth: true
 * });
 * ```
 */
```

### Success Criteria for Migration Documentation

1. Clarity
- Every new component has clear documentation of its final intended form
- All temporary compatibility code is marked with @removal-target
- Migration paths are explicitly documented
- Each phase has clear documentation of what can be removed

2. Testability
- Migration status can be verified programmatically
- Tests exist to verify both old and new systems
- Clear metrics for migration progress

3. Maintainability
- Documentation makes it clear what is temporary
- Removal targets are specific and verifiable
- Migration utilities are themselves temporary and marked for removal

4. Confidence
- Each phase has explicit verification steps
- Migration progress can be measured
- Regressions can be caught early

=== ADDITION 2 

─────────────────────────────────────────────────────────────────────────
3) MIGRATION AND DOCUMENTATION STRATEGY
─────────────────────────────────────────────────────────────────────────

The instrumentation layer must be introduced carefully to avoid disrupting our 504 passing tests. Every component should be documented with both its final intended form and clear migration paths.

A) Documentation Requirements
   • Every new component must have JSDoc with @package and @remarks explaining its final intended form
   • All temporary compatibility code must be marked with @deprecated and @removal-target
   • Migration paths must be explicit and testable
   • Each phase's removable components must be clearly identified

B) Migration Utilities
   • Introduce StateMigrationHelper to assist test migration
   • Support running tests through both old and new systems
   • Track migration progress programmatically
   • Self-mark as temporary with @internal and @removal-target

C) Verification Requirements
   • Each migration step must be independently verifiable
   • Tests must exist for both old and new paths during migration
   • Clear metrics must show migration progress
   • Regressions must be detectable early

D) Code Organization
   • New instrumentation code in state/instrumentation/
   • Compatibility layer in state/legacy/ (marked for removal)
   • Migration utilities in state/migrations/ (temporary)
   • Tests organized to separate pure new system from migration tests

This strategy ensures we can:
• Maintain control during migration
• Clearly identify temporary code
• Have a verifiable path to removing old code
• Help future maintainers understand what's temporary
• Measure migration progress programmatically