meld

Below is a consolidated list of concrete changes needed to make the architecture design documents cohesive and to align the codebase with the new, services-based approach. It also specifies which existing code/tests should be kept, modified, or deleted. There is no need for backward compatibility, so we can remove or refactor anything that does not serve the updated design. ───────────────────────────────────────────────────────────────────────── 1. Consolidate and Rename Folders to Match the Services-Based Layout [IN PROGRESS] ───────────────────────────────────────────────────────────────────────── ✓ Create "services" folder at project root. Inside it: ✓ ParserService/ ✓ InterpreterService/ ✓ DirectiveService/ ✓ ValidationService/ ✓ StateService/ ✓ PathService/ ✓ FileSystemService/ ✓ OutputService/ • ResolutionService/ ✓ Within each service folder, have exactly: ✓ A main service file (e.g., PathService.ts) ✓ A matching test file (e.g., PathService.test.ts) ✓ (Optional) a small README or subfiles if the service is complex • [TODO] Remove/distribute "utils.ts,", "fs.ts,", "fs-extra.ts,", "fs-promises.ts,", "fs-utils.ts,", etc.: – Merge their logic into FileSystemService (or a single FileSystemAdapter) if needed. – Delete any leftover or redundant FS-mocking code once FileSystemService covers it. • [TODO] Move the "validators" subfolder inside ValidationService (e.g. "ValidationService/validators/"), as described in the design docs. • [TODO] Eliminate the "__mocks__" directories if redundant. The new FileSystemService or test adapters should replace them. ───────────────────────────────────────────────────────────────────────── 2. Adopt the New "ParserService" to Call meld-ast [COMPLETED] ───────────────────────────────────────────────────────────────────────── ✓ In "services/ParserService" folder, created ParserService.ts that: ✓ Wraps meld-ast's parse() function. ✓ Removes all direct usage of parseMeld in random places. ✓ Added proper error handling with MeldParseError. ✓ Added location tracking support. ✓ Created comprehensive test suite. • Delete or refactor the old "meld-ast.ts" file that duplicates parse logic: – Keep the meld-ast library references and type definitions, but remove any custom code we wrote that partially replicates meld-ast's behavior. • Remove the parse methods from "parser.ts" in the root if they conflict or are redundant. Migrate needed logic into ParserService. ───────────────────────────────────────────────────────────────────────── 3. Introduce "InterpreterService" for Node Iteration [80% -- WAITING ON DIRECTIVES TO BE COMPLETED] ───────────────────────────────────────────────────────────────────────── [See detailed design in service-interpreter.md] • Create "InterpreterService.ts" (or rename "interpreter/interpreter.js"): – Orchestrates reading each MeldNode (Text vs. Directive), then calling DirectiveService. – Cleans up old references to "interpret" scattered in "interpreter.test.ts" or "meld.ts." • Keep relevant tests from "interpreter.test.ts" but re-scope them to the new InterpreterService class. Delete any legacy bits referencing old partial logic we no longer need. ───────────────────────────────────────────────────────────────────────── 4. Create "DirectiveService" with a Registry + Handlers [PARTIAL -- SEE BELOW] ───────────────────────────────────────────────────────────────────────── • In "DirectiveService/" folder: – A main DirectiveService.ts that routes DirectiveNode → correct handler (Text, Data, Import, etc.). – A "handlers/" subfolder with each directive's logic: • e.g., TextDirectiveHandler.ts, DataDirectiveHandler.ts, EmbedDirectiveHandler.ts, ImportDirectiveHandler.ts, etc. • Move the existing directive logic from "define.ts," "data.ts," "text.ts," "embed.ts," "import.ts," "run.ts," etc. into these smaller directive handler files. • Delete old "cmd.ts," "cli.test.ts," or "cli.md" references if they are specific to the old CLI approach: – Or, if we still want a CLI, keep a small "cli/" folder that just calls the new services. ───────────────────────────────────────────────────────────────────────── 5. Add a "ValidationService" for Directive Checking [COMPLETED] ───────────────────────────────────────────────────────────────────────── ✓ Created ValidationService with: ✓ Core validators for text, data, import, and embed directives ✓ Extensible validator registration system ✓ Proper error handling with location information ✓ Comprehensive test coverage ✓ Removed partial validations from directive handlers ✓ Consolidated validation logic into dedicated validators ✓ Added proper logging integration ───────────────────────────────────────────────────────────────────────── 6. Migrate Variable & State Logic into "StateService" [COMPLETED] ───────────────────────────────────────────────────────────────────────── ✓ Create "StateService/StateService.ts" with the structure shown in the design docs: ✓ Raw variable storage (textVars, dataVars, pathVars, commands) ✓ No resolution logic (moved to ResolutionService) ✓ createChildState() for nested @import or @embed usage ✓ State hierarchy management ✓ Remove "state.ts," "state.test.ts," or "interpreter/state/state.js" references ✓ Migrated the best parts of "interpreter/state/state.js" into "StateService.ts" ───────────────────────────────────────────────────────────────────────── 7. Create "ResolutionService" for Variable Resolution [90% COMPLETE -- WAITING ON DIRECTIVES TO BE COMPLETED] ───────────────────────────────────────────────────────────────────────── ✓ Created ResolutionService with: ✓ Variable resolution (${var}, #{data}, $path) ✓ Command resolution ($command(args)) ✓ Path resolution in various contexts ✓ Resolution context management ✓ Explicit cycle detection for variables ✓ Context validation rules ✓ Added dedicated resolvers: ✓ TextResolver (with nested interpolation detection) ✓ DataResolver (with field access validation) ✓ PathResolver (enforcing $HOMEPATH/$PROJECTPATH) ✓ CommandResolver (validating parameter types) ✓ Added ResolutionContextFactory: ✓ Pre-defined contexts for each directive type ✓ Enforces grammar rules per context ✓ Prevents invalid variable usage ✓ Implemented clear separation between: ✓ Variable reference cycles (in ResolutionService) ✓ File import cycles (in CircularityService) • [TODO] Add remaining edge case tests • [TODO] Complete integration tests with DirectiveService ───────────────────────────────────────────────────────────────────────── 8. Update DirectiveService for New Architecture [90% COMPLETE] ───────────────────────────────────────────────────────────────────────── ✓ Basic service structure and interfaces ✓ Handler registration system ✓ Complete directive handlers: ✓ Definition handlers: ✓ @text ✓ @data ✓ @path ✓ @define ✓ Execution handlers: ✓ @run ✓ @embed ✓ @import ✓ Update handlers to: ✓ Store raw values in StateService ✓ Use ResolutionService for all resolution ✓ Use ResolutionContextFactory for correct contexts ✓ Handle resolution errors properly ✓ Add proper error handling: ✓ Validation errors from ValidationService ✓ Resolution errors from ResolutionService ✓ Execution errors from handlers ✓ Update tests to verify: ✓ Correct context usage ✓ Error handling ✓ Integration with ResolutionService REMAINING TASKS: • Final integration testing with all services working together • Edge case testing with complex scenarios • Performance optimization if needed ───────────────────────────────────────────────────────────────────────── 9. Implement InterpreterService for Node Iteration [80% COMPLETE] ───────────────────────────────────────────────────────────────────────── ✓ Created InterpreterService with: ✓ Node iteration and processing ✓ State management ✓ Basic error handling ✓ Integration with DirectiveService ✓ Comprehensive test coverage for basic functionality • [TODO] Add advanced error handling • [TODO] Add edge case handling • [TODO] Complete integration tests with all services ───────────────────────────────────────────────────────────────────────── 10. Create CLIService for Command Line Interface [70% COMPLETE] ───────────────────────────────────────────────────────────────────────── ✓ Basic CLI implementation ✓ Argument parsing ✓ Test coverage for main functionality • [TODO] Improve error handling • [TODO] Add more format options • [TODO] Enhance logging • [TODO] Add configuration options ───────────────────────────────────────────────────────────────────────── 11. Implement a Proper "PathService" [COMPLETED] ───────────────────────────────────────────────────────────────────────── ✓ Created PathService with: ✓ Path resolution and validation ✓ Variable expansion ($HOME, $PROJECTPATH, etc.) ✓ Base directory restrictions ✓ File/directory type validation ✓ Comprehensive test coverage ✓ Removed/distributed old path utilities: ✓ Consolidated path.ts, location.ts into PathService ✓ Migrated test coverage from path.test.ts ───────────────────────────────────────────────────────────────────────── 12. Create a Proper "FileSystemService" [COMPLETED] ───────────────────────────────────────────────────────────────────────── ✓ Created FileSystemService with: ✓ Core file operations (readFile, writeFile, exists, stat) ✓ Directory operations (readDir, ensureDir, isDirectory) ✓ Path operations (join, resolve, dirname, basename) ✓ Test mode with mocking capabilities ✓ Comprehensive test coverage ✓ Removed/distributed old FS utilities: ✓ Consolidated fs.ts, fs-extra.ts, fs-promises.ts, fs-utils.ts into FileSystemService ✓ Migrated test coverage from fs-utils.test.ts ───────────────────────────────────────────────────────────────────────── 13. "OutputService" to Handle Markdown vs. LLM XML ───────────────────────────────────────────────────────────────────────── [See detailed design in service-output.md] • Move "converter.ts," "llmxml-utils.ts," or "toLLMXml" references into "OutputService/formats/LLMOutput.ts." • Keep essential tests that confirm LLM XML usage from "llmxml-utils.test.ts." • For markdown output, keep "toMarkdown" logic in "OutputService/formats/MarkdownOutput.ts." • The new OutputService just picks the correct converter: – e.g., outputService.convert(nodes, state, 'md' | 'llm') • Remove "CONVERTER.md" or unify it into a short doc in "OutputService/README.md" if desired. ───────────────────────────────────────────────────────────────────────── 14. Remove Or Rework "runMeld," "meld.ts," "cli.test.ts," Etc. ───────────────────────────────────────────────────────────────────────── • The old "runMeld" function in "meld.ts" or "cli.md" lumps together parse, interpret, and formatting in a single step. Under the new design: – We can keep a top-level "sdk/index.ts" that offers a unify function runMeld() if we want a single call. But internally, it calls: 1) parserService.parse 2) interpreterService.interpret 3) outputService.convert – Or we remove "runMeld" entirely if the new design doesn't want that monolithic approach. • "cli.test.ts," "cmd.ts," or "CLI.md" can be removed if we do not need a CLI. If we do keep a CLI, it should just be a thin wrapper calling the new services. ───────────────────────────────────────────────────────────────────────── 15. Clean Up Tests: Keep the Good Coverage, Delete Redundancies ───────────────────────────────────────────────────────────────────────── • Organize tests into "tests/unit/ServiceName.test.ts" and "tests/integration/...". • Keep coverage from "embed.test.ts," "import.test.ts," "data.test.ts," etc. if they reflect real directive logic. Move them into "DirectiveService/handlers/tests/" or "integration." • Remove partial or outdated tests that no longer correspond to the new services. • Maintain "Integration" tests that spin up the entire pipeline (ParserService → InterpreterService → OutputService). ───────────────────────────────────────────────────────────────────────── 16. Merge or Delete Partial or Duplicate Implementation Files ───────────────────────────────────────────────────────────────────────── • "location-helpers.ts," "location.ts," or "error-locations.test.ts" can be merged if we only need a single place to handle location adjustments. • "meld-spec.d.ts" and "meld-ast.d.ts" can remain if they supply needed extra definitions but remove any partial duplication of official meld-ast or meld-spec. • The "grammar" docs remain as reference, but integrate them into the new doc folder or remove references to old partial grammar code. ───────────────────────────────────────────────────────────────────────── 17. Update The "DESIGN DOCS TO BE REVIEWED AND IMPROVED" ───────────────────────────────────────────────────────────────────────── • Rewrite them to use consistent references to the new directory layout (services/ParserService, etc.). • Remove repeated or contradictory sections that mention old code we are deleting. • Reorder them so each service doc (PathService, FileSystemService, ValidationService, etc.) matches the final structure. ───────────────────────────────────────────────────────────────────────── 18. Simplify The Logging Approach [COMPLETED] ───────────────────────────────────────────────────────────────────────── ✓ Created a centralized "logger.ts" in "core/utils/" using Winston. ✓ Unified all logging into service-specific loggers with proper configuration. ✓ Added structured logging with file and console output. ───────────────────────────────────────────────────────────────────────── 19. Final Clarifications ───────────────────────────────────────────────────────────────────────── • No backward compatibility means we are free to remove everything not used in the final approach. • The new architecture is fully services-based: – No partial leftover "run logic" or "fs logic" in random utility files. • Move forward with SOLID design: each directive in a separate file, each service tested in isolation. ───────────────────────────────────────────────────────────────────────── NEXT STEPS ───────────────────────────────────────────────────────────────────────── The next services to implement in order of dependency should be: 1. ResolutionService - This is needed by DirectiveService for all variable resolution • Start with core resolvers (Text, Data, Path, Command) • Add ResolutionContextFactory • Implement cycle detection • Add context validation 2. Updated StateService - This needs to be modified to focus on raw storage 3. Updated DirectiveService - This needs to be reorganized to use ResolutionService • Use ResolutionContextFactory • Update error handling 4. ValidationService - This is needed by DirectiveService for validating directives 5. InterpreterService - This orchestrates the overall interpretation process 6. OutputService - This handles the final output formatting This order ensures we build from the ground up, with each service having its dependencies available. ───────────────────────────────────────────────────────────────────────── CONCLUSION ───────────────────────────────────────────────────────────────────────── By applying these 19 sets of changes, we will create a cohesive, maintainable codebase that strictly follows the services-based design described in the updated architecture documents. This yields a simpler code layout, targeted tests, and a clear separation of concerns for Parsing, Interpreting, Validating, Managing State, Handling Directives, Resolving Paths, Reading Files, and Outputting results.