@meldscience/meld/docs/PLAN-2a.md

pipeable one-shot prompt scripting toolkit

Below is a proposed path to migrate away from custom regex parsing toward a unified/remark/mdast-based approach, including:
	1.	High-level architectural approach
	2.	Concrete implementations for the tricky pieces (i.e., where less experienced developers are likely to err)
	3.	Test changes
	4.	Detailed plan for how an AI (or any developer) can implement and integrate everything else.

1. High-Level Architectural Approach

A. Stop using regex for @cmd[...] and @import[...]

All current code for extracting placeholders and their parameters (in extractPlaceholders.ts) will be replaced by a remark plugin that:
	•	Reads the Markdown AST.
	•	Looks for inline “directives” or “phrases” that match @cmd[...] or @import[...].
	•	Replaces them (in the AST) with a placeholder “node” containing the relevant metadata (command name, file path, heading overrides, symbols, etc.).

B. Single Source of Truth in the AST

Instead of string-splitting or searching line-by-line, we will rely on remark to parse the entire Markdown. Then we can:
	1.	Walk the AST, identify the custom “@cmd” and “@import” directives.
	2.	Convert them into either:
	•	Custom MDAST nodes (like a meldDirective node).
	•	Or a form of remark directive (using remark-directive), if you want to treat them as “leaf directives” or “text directives.”

C. Downstream Processing

After we have the placeholders as well-defined AST nodes:
	•	For each @cmd[...] node, run the command, capture the stdout, and replace the AST node’s contents with that output (potentially as a code block or raw text).
	•	For each @import[...] node, we do the same sort of logic we have now in separate modules (like markdownImporter or codeImporter), but with the difference that the final content is placed back into the AST.

D. Final Output

Finally, once we have a fully updated AST (with all placeholders replaced), we:
	•	Re-stringify (using remark-stringify or a similar library) to produce the final markdown output.
	•	Return that from meld.

E. Breaking Up the Code

We can keep each major step in its own file or plugin:
	1.	remarkMeldDirectives.ts: Contains the remark plugin for scanning the AST for @cmd[...] and @import[...].
	2.	remarkMeldCmd.ts: A plugin (or routine) that, once we have nodes for commands, executes them.
	3.	remarkMeldImports.ts: A plugin (or routine) that, for each @import[...] node, uses markdownImporter or codeImporter to embed the content.

…and so on. The key point is to keep each concern separate.

2. Tricky, Complex Parts (with Example Implementations)

Below is code for (a) the remark plugin that identifies @cmd[...] / @import[...] within text, and (b) a function that processes them. These are the parts “most likely to trip up less intelligent code authors.”

	NOTE: We assume you’ve installed the needed packages:

2A. remarkMeldDirectives.ts (core plugin to parse @cmd[...] and @import[...])

import { Plugin } from 'unified';
import { visit } from 'unist-util-visit';
import { Node } from 'unist';
import { parseMeldDirective } from './parseMeldDirective';

/**
 * Our custom node type for meld placeholders. 
 * 
 * You could define them more thoroughly in a .d.ts or types.ts file:
 */
export interface MeldDirectiveNode extends Node {
  type: 'meldDirective';
  meldType: 'cmd' | 'import';
  raw: string;            // the full matched directive string
  lineNumber?: number;    // for error reporting if wanted
  data?: {
    // We'll store parse results here:
    command?: string;
    importPath?: string;
    heading?: string;
    importSymbol?: string;
    importSymbolList?: string[];
    headingLevelOverride?: string;
    headingTextOverride?: string;
  };
}

/**
 * remarkMeldDirectives: a remark plugin that looks for textual tokens like `@cmd[...]` or `@import[...]`
 * and converts them into a custom AST node with structured data.
 */
export const remarkMeldDirectives: Plugin = function remarkMeldDirectives() {
  return (tree, file) => {
    // We'll walk the AST, looking specifically at "text" nodes.
    visit(tree, 'text', (node: any, index: number | null, parent: any) => {
      if (!node.value) return;

      // We'll parse out all meld directives from node.value.
      const newChildren: any[] = parseMeldDirective(node.value, file);

      if (newChildren.length > 1 || (newChildren.length === 1 && newChildren[0].type !== 'text')) {
        // Replace the original text node with the new set of (text + meldDirective) nodes
        parent.children.splice(index, 1, ...newChildren);
        return index + newChildren.length;
      }
      return;
    });
  };
};

2B. parseMeldDirective.ts (the function that does the actual string-level parse**—**the tricky part)

import { MeldDirectiveNode } from './remarkMeldDirectives';
import { Text } from 'mdast'; // from @types/mdast

/**
 * Splits a line of text into an array of either:
 *   - raw text nodes
 *   - meldDirective nodes
 *
 * This is the biggest piece of “regex-like” logic, 
 * but we keep it minimal: just enough to detect the pattern
 * and gather content for a MeldDirectiveNode. 
 */
export function parseMeldDirective(value: string, file: any): Array<MeldDirectiveNode | Text> {
  const result: Array<MeldDirectiveNode | Text> = [];
  let remaining = value;

  const directiveRegex = /(@cmd\[.*?\]|@import\[.*?\])/g;

  let match: RegExpExecArray | null;
  let lastIndex = 0;

  while ((match = directiveRegex.exec(remaining)) !== null) {
    const start = match.index;
    const end = start + match[0].length;

    // Add any preceding text as a Text node
    if (start > lastIndex) {
      const textChunk = remaining.slice(lastIndex, start);
      result.push({ type: 'text', value: textChunk });
    }

    // Build the directive node
    const rawDirective = match[0];
    const directiveNode: MeldDirectiveNode = buildMeldDirectiveNode(rawDirective);
    result.push(directiveNode);

    lastIndex = end;
  }

  // Add any trailing text
  if (lastIndex < remaining.length) {
    const textChunk = remaining.slice(lastIndex);
    result.push({ type: 'text', value: textChunk });
  }

  return result;
}

/**
 * Helper that parses the `@cmd[...]` or `@import[...]` line and returns a typed node.
 */
function buildMeldDirectiveNode(raw: string): MeldDirectiveNode {
  // We do minimal checks, because big logic belongs in codeImporter or markdownImporter.
  if (raw.startsWith('@cmd')) {
    const command = raw.slice('@cmd['.length, -1).trim();

    return {
      type: 'meldDirective',
      meldType: 'cmd',
      raw,
      data: {
        command
      }
    };
  } else {
    // It's an @import directive
    // Example: "@import[src/meld.ts # {type}] as ## 'Blah'" 
    // or simpler: "@import[file#Heading]"
    // We'll do enough parsing to store partial info in data
    // Then we rely on existing codeImporter/markdownImporter logic to do deeper checks.
    const inside = raw.slice('@import['.length, -1).trim();
    // We'll store the entire inside as something for the importer to parse
    // or do partial splitting:
    // e.g. parse out file vs heading or code symbol
    // For brevity, do something very simple:

    let importPath = inside;
    let heading: string | undefined;

    if (inside.includes('#')) {
      const [pathPart, headingPart] = inside.split('#');
      importPath = pathPart.trim();
      heading = headingPart.trim();
    }

    return {
      type: 'meldDirective',
      meldType: 'import',
      raw,
      data: {
        importPath,
        heading
      }
    };
  }
}

	Why is this the “tricky” part?
		•	Off-by-one errors in string slicing
	•	Handling multiple placeholders on the same line
	•	Correctly capturing text before and after placeholders
	•	Deciding how to store partial parse data

2C. processMeldNodes.ts (the plugin that processes each directive node, calling your existing import or command logic)

import { Plugin } from 'unified';
import { visit } from 'unist-util-visit';
import { MeldDirectiveNode } from './remarkMeldDirectives';
import { importCodeSymbols } from '../codeImporter';
import { importMarkdownSection } from '../markdownImporter';
import { execSync } from 'child_process';

export interface ProcessMeldNodesOptions {
  currentFileDir: string; // so we can resolve relative paths
}

export const remarkProcessMeldNodes: Plugin<[ProcessMeldNodesOptions]> = function remarkProcessMeldNodes(options) {
  return async (tree, file) => {
    const errors: string[] = [];

    visit(tree, 'meldDirective', (node: MeldDirectiveNode, index, parent) => {
      if (node.meldType === 'cmd') {
        try {
          const output = execSync(node.data?.command || '', { encoding: 'utf-8' }).trim();
          // Replace with a text node containing the output
          parent.children[index] = {
            type: 'text',
            value: output
          };
        } catch (error: any) {
          // Insert an error code block or similar
          parent.children[index] = {
            type: 'text',
            value: `Command produced error output:\n\`\`\`\n${error.message}\n\`\`\``
          };
        }
      } else {
        // It's an import directive
        const importPath = node.data?.importPath;
        if (!importPath) {
          // Replace with an error text
          parent.children[index] = {
            type: 'text',
            value: 'Invalid import directive: missing import path'
          };
          return;
        }

        // Logic to figure out if it's a .md or .ts
        if (importPath.endsWith('.md')) {
          const heading = node.data?.heading;
          // Reuse existing markdownImporter
          try {
            const result = importMarkdownSection({
              filePath: /* resolve with options.currentFileDir + importPath, etc. */,
              heading,
              headingLevelOverride: node.data?.headingLevelOverride,
              headingTextOverride: node.data?.headingTextOverride
            });
            if (!result.headingFound) {
              parent.children[index] = {
                type: 'text',
                value: `Heading "${heading}" not found in ${importPath}`
              };
            } else if (result.duplicateHeadings) {
              parent.children[index] = {
                type: 'text',
                value: `Multiple headings "${heading}" found in ${importPath}`
              };
            } else {
              // Insert the imported text as a new child
              parent.children[index] = {
                type: 'text',
                value: result.content
              };
            }
          } catch (err: any) {
            parent.children[index] = {
              type: 'text',
              value: `Error importing from ${importPath}: ${err.message}`
            };
          }
        } else {
          // It's a code file
          // You might parse out symbol(s) from node.data
          const symbols = node.data?.importSymbolList 
            ?? (node.data?.importSymbol ? [node.data?.importSymbol] : []);
          try {
            const result = importCodeSymbols({
              filePath: /* resolve full path */,
              symbolList: symbols,
              headingLevelOverride: node.data?.headingLevelOverride,
              headingTextOverride: node.data?.headingTextOverride
            });
            if (!result.symbolsFound) {
              parent.children[index] = {
                type: 'text',
                value: `One or more symbols not found in ${importPath}`
              };
            } else if (result.duplicateSymbols) {
              parent.children[index] = {
                type: 'text',
                value: `Duplicate symbols found in ${importPath}`
              };
            } else {
              parent.children[index] = {
                type: 'text',
                value: result.content
              };
            }
          } catch (err: any) {
            parent.children[index] = {
              type: 'text',
              value: `Error importing from ${importPath}: ${err.message}`
            };
          }
        }
      }
    });

    // Optionally attach errors on file.data if you want
    (file.data as any).meldErrors = errors;
  };
};

These 2–3 plugins together are the critical, complicated glue. The rest is just hooking them into the pipeline (via unified(), .use(remarkMeldDirectives), .use(remarkProcessMeldNodes, { ... }), .use(remarkStringify) etc.).

3. Changes Required to the Tests

A. extractPlaceholders.test.ts
	•	This test is basically replaced by the new AST plugin approach.
	•	Instead of calling extractPlaceholders(content), you will run unified().use(remarkParse).use(remarkMeldDirectives) ... .runSync(tree) or .processSync(content) and then check the resulting AST.
	•	Expect that you have a set of meldDirective nodes in the tree.
	•	You can test the shape of those nodes (like meldType === 'cmd').
	•	Alternatively, you can keep the same “public API” name but route it through the remark pipeline.

B. meld.test.ts
	•	Instead of processImports, your top-level function might do something like:
	1.	const file = await unified().use(remarkParse).use(remarkMeldDirectives).use(remarkProcessMeldNodes).use(remarkStringify).process(content)
	2.	Check file.data.meldErrors for errors
	3.	file.value is your processed markdown.
	•	The tests that expect lines like:

expect(result.content).toBe('...')
expect(result.errors).toEqual([...])

become:

expect(String(file)).toBe('...')
expect(file.data.meldErrors).toEqual([...])


	•	Also note that “multiple placeholders on the same line” logic can be enforced in the plugin if you wish. If you still want to disallow that, you can do so in parseMeldDirective.ts by checking if you found > 1 placeholders in a single text node.

C. markdownImporter.test.ts, codeImporter.test.ts
	•	These can remain mostly the same because they test the logic of extracting the correct content from code or markdown files. They do not rely heavily on the old extractPlaceholders (they only test the direct function calls).

D. oneshotcat.test.ts, oneshot.test.ts
	•	Currently they rely on processImports. Now you’ll rely on unified + remark plugins + remarkStringify to produce the final content with commands executed. So you must update them to:
	•	Possibly call a new meldProcessFile(...) that does the pipeline. Or call it inline with unified.
	•	Check the final .value for embedded command output or imported content.

E. CLI Tests (like meld-config.test.ts)
	•	These appear mostly unaffected because they test the config functionality, not the placeholder logic. Possibly zero changes unless you rename the “meld” CLI.

4. Extremely Detailed Implementation Plan

Below is a step-by-step plan that an AI or developer can follow to implement everything around these new core changes.
	1.	Install the required libraries

npm install unified remark-parse remark-stringify unist-util-visit remark-directive


	2.	Create/Refactor plugin files
	•	plugins/remarkMeldDirectives.ts
	•	Contains the main AST transform to detect @cmd[...] and @import[...] tokens within text nodes.
	•	plugins/parseMeldDirective.ts
	•	Does the partial string-level parse for each match.
	•	Yields an array of AST nodes (some text, some meldDirective).
	•	plugins/remarkProcessMeldNodes.ts
	•	Contains logic that visits each meldDirective node and calls importMarkdownSection, importCodeSymbols, or runs the shell command.
	3.	Replace extractPlaceholders.ts
	•	The entire file can be removed or replaced by the new plugin.
	•	If needed, keep a small function extractPlaceholders that internally does:

export async function extractPlaceholders(content: string) {
  const file = await unified()
    .use(remarkParse)
    .use(remarkMeldDirectives)
    .process(content);

  // Walk the AST nodes in file to gather all `meldDirective`s
  // Return them in a structured array.
}


	•	This is optional if you want to preserve the old function name for the test harness, but it is not strictly necessary.

	4.	Rewrite meld.ts
	•	processImports is replaced with a function that runs the pipeline:

import { unified } from 'unified';
import remarkParse from 'remark-parse';
import remarkStringify from 'remark-stringify';
import { remarkMeldDirectives } from './plugins/remarkMeldDirectives';
import { remarkProcessMeldNodes } from './plugins/remarkProcessMeldNodes';

export async function processMeldFile(inputPath: string, workspacePath: string): Promise<{ content: string, errors: string[] }> {
  // read the file
  const content = fs.readFileSync(inputPath, 'utf8');
  
  // run pipeline
  const file = await unified()
    .use(remarkParse)
    .use(remarkMeldDirectives)
    .use(remarkProcessMeldNodes, { currentFileDir: path.dirname(inputPath) })
    .use(remarkStringify)
    .process(content);

  const meldErrors = (file.data as any).meldErrors || [];
  return {
    content: String(file),
    errors: meldErrors
  };
}


	•	Update Meld.process() to call processMeldFile(...).

	5.	Tests
	•	extractPlaceholders.test.ts
	•	Replace with “test the plugin approach.” Possibly rename to remarkMeldDirectives.test.ts.
	•	Use unified().use(remarkParse).use(remarkMeldDirectives), pass in the string, and then visit the AST to confirm the presence or shape of the meldDirective nodes.
	•	meld.test.ts
	•	Point it at the new pipeline code. Then read the final .content and check .errors.
	•	Where it used to check “multiple placeholders on the same line,” you can either keep or remove that constraint. If you keep it, implement a check in parseMeldDirective.ts that errors if more than one match is found in the same chunk.
	•	Others remain mostly the same, except that references to the old function name or usage might need a tweak.
	6.	Incrementally Remove Old Code
	•	Once the new plugins are stable, remove extractPlaceholders.ts, any direct “regex-based placeholder extraction,” and unify any leftover logic inside the new plugin.
	•	markdownImporter.ts and codeImporter.ts can remain, but they can eventually be simplified or improved if you choose to parse code blocks with actual AST libraries for TypeScript or remark sections for markdown.
	7.	Validate & Release
	•	Run all tests, ensure coverage is good.
	•	Confirm that the new pipeline supports your needed features: heading level overrides, @cmd, etc.

Closing Note

With this approach, you gain:
	•	A robust AST-based pipeline that’s easy to extend or customize.
	•	The ability to handle tricky corner cases (like nested backtick code blocks, or leading/trailing text) in a consistent manner.
	•	Cleaner separation of concerns: each plugin deals with a single concern (detecting placeholders vs. fulfilling them).

You lose:
	•	The inherent complexity that came from heavy string-level logic in multiple places.

That’s a good tradeoff!