mlld/CHANGELOG.md

mlld: llm scripting language

# Changelog

All notable changes to the mlld project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [2.0.0-rc81]

### Added
- **Self-documenting help system**: `mlld howto` provides LLM-accessible documentation directly in the CLI
  - `mlld howto` - Show topic tree with intro pinned at top
  - `mlld howto intro` - Introduction with mental model and key concepts
  - `mlld howto <topic>` - Show all help for a specific topic (e.g., `mlld howto when`)
  - `mlld howto <topic> <subtopic>` - Show specific subtopic (e.g., `mlld howto when first`)
  - `mlld howto grep <pattern>` - Search across all atoms for matching lines
  - `mlld qs` / `mlld quickstart` - Quick start guide
  - Built on atom-based documentation architecture (docs/src/atoms/) enabling DRY content reuse
  - Pattern documented in docs/dev/HOWTO-PATTERN.md for adoption by other tools
  - Documentation atoms: 106 atoms extracted covering intro, syntax, commands, modules, patterns, security, configuration, and common mistakes
  - Git pre-commit hook auto-updates atom 'updated' dates when modified atoms are staged
  - Colorized terminal output: syntax-highlighted code blocks, colored headers, topic tree with colored categories and IDs
- **Prose execution**: Define executable functions that invoke a prose interpreter via LLM
  - Syntax: `exe @fn(params) = prose:@config { inline content }`
  - File-based: `exe @fn(params) = prose:@config "file.prose"`
  - Template files: `.prose.att` (`@var`) and `.prose.mtt` (`{{var}}`); `.prose` files do not interpolate
  - Config uses model executors: `{ model: @opus, skills: [...] }`
  - Pre-built configs available from `@mlld/prose` public module
  - Requires [OpenProse](https://prose.md) skill or another prose interpreter
  - Skills must be approved in Claude Code before use
  - See [docs/user/prose.md](docs/user/prose.md) for full documentation
- **`mlld validate`**: Static analysis command for syntax validation without execution
  - `mlld validate <file>` - Validate syntax and show module structure (exports, imports, executables, guards, needs)
  - `--format json` - Machine-readable JSON output for tooling integration
  - `--ast` - Include parsed AST in JSON output (requires `--format json`)
  - Returns exit code 1 for invalid files, enabling CI/toolchain integration
  - `mlld analyze` as alias

### Changed
- **Terminology**: "prose mode" renamed to "markdown mode" to avoid confusion with prose execution
  - `.md` and `.mld.md` files use "markdown mode" (slash-prefixed directives)
  - "prose" now refers to OpenProse/prose execution, not the file format

### Fixed
- **`mlld howto` shows all atom categories**: Fixed howto command to load atoms from all 8 categories (syntax, commands, control-flow, modules, patterns, configuration, security, mistakes) instead of only control-flow

## [2.0.0-rc80]

### Added
- **`mlld docs @author/module`**: New command to display module documentation, showing `# tldr` section followed by `# docs` section from published modules
- **`mlld info` shows real registry data**: Now fetches actual module metadata from registry (version, needs, license, repo, keywords) and appends the `# tldr` section
- **Colorized CLI output**: Module info and docs display with syntax highlighting for mlld code blocks, colored headers, and formatted metadata
- **`mlld-run` blocks use strict mode**: Code inside `mlld-run` fenced blocks now parses in strict mode (slashes optional, no prose between directives)
- **Transitive dependency installation**: `mlld install` now automatically installs mlld module dependencies (npm-style)
  - When installing `@alice/utils`, its `dependencies` from frontmatter are discovered and installed
  - Recursive: transitive deps of deps are also installed
  - All modules recorded in `mlld-lock.json` (config unchanged - only direct deps in `mlld-config.json`)
  - Install summary shows breakdown: "3 modules installed (1 direct, 2 transitive)"
  - Lazy runtime fetching still works as fallback for modules not pre-installed

### Fixed
- **Module publish validation for exe declarations**: Fixed `ExportValidator` not recognizing exe declarations where the identifier is a `VariableReference` node. This caused `mlld publish` to fail with "Exported name is not declared" errors for modules like `@mlld/array` and `@mlld/string`.
- **Module scope isolation for nested imports**: Fixed bug where importing a module that internally imports from another module would cause "variable already imported" errors. Child module scopes are now properly isolated from parent scope during import evaluation.
- **Executable preservation in object properties**: Fixed bug where executables stored as object properties would lose their Variable wrapper during import, causing `isExecutableVariable()` to return false. Object property executables are now properly reconstructed during import.
- **Registry publish SHA error**: Fixed "sha wasn't supplied" error when publishing new versions of existing modules. The existing tags.json SHA is now properly fetched and provided when updating.
- **Duplicate version publish check**: `mlld publish` now checks if the specific version already exists in the registry before attempting to create a PR, preventing wasted effort on duplicate publishes.

## [2.0.0-rc79]

### Added
- **Strict mode for .mld files**: Bare directive syntax without slash prefixes
  - `.mld` files use strict mode: bare directives (`var`, `show`, `exe`), text lines error, blank lines ignored
  - `.mld.md` and `.md` files use markdown mode: require `/` prefix, text becomes content (existing behavior)
  - Slash prefix optional in strict mode for backward compatibility (`/var` and `var` both work)
  - SDK defaults to strict mode for raw strings (no file path) and unknown extensions
  - File extension determines parsing mode: `.mld` → strict, `.mld.md`/`.md` → markdown
  - CLI flags: `--loose`/`--markdown`/`--md`/`--prose` force markdown mode
  - Mode included in AST cache keys to differentiate same file parsed in different modes
- **Block syntax for exe and for**: Multi-statement bodies using `[...]` delimiters
  - Exe blocks: `/exe @func() = [let @x = 1; let @y = 2; => @x + @y]` (statements separated by newlines or semicolons)
  - For blocks: `/for @item in @items [show @item; let @count += 1]` (`=>` optional for block bodies)
  - When-expression exe block actions: `when first [ @cond => [...] ]` supports full exe block semantics (let, side effects, return)
  - `let @var = value` creates block-scoped variables; `let @var += value` for augmented assignment (arrays, strings, objects, numbers)
  - `=> @value` optional return must be the last statement when present in exe blocks
  - Nested for/when inside blocks supported; inner directives are slashless
  - Blocks use `[...]` (not `{...}`) to distinguish mlld control flow from code/command/data
- **Block syntax for var**: `/var @value = [let @x = ...; => @x]` evaluates a local statement block and returns its value
- **Block comments in bracket bodies**: `>>`/`<<` comments inside `[...]` blocks (exe, for, when, guard, when-expressions) are consumed as whitespace
- **While loops**: Bounded iteration with `done`/`continue` control flow
  - `/while (100) @processor` - directive form with iteration cap
  - `@input | while(100, 1s) @processor` - pipeline stage with optional pacing
  - `done @value` terminates iteration and returns value
  - `continue @newState` advances to next iteration with new state
  - `@mx.while.iteration` (1-based) and `@mx.while.limit` available in processor
- **Streaming format adapters**: NDJSON streaming parsed via configurable adapters
  - `with { streamFormat: "claude-code" }` for Claude SDK-specific parsing
  - Default `ndjson` adapter handles generic JSON streaming
  - `with { streamFormat: @adapterConfig }` accepts custom AdapterConfig objects
- **Command working directories**: `cmd:/abs/path`, `sh:/abs/path`, `bash:/abs/path`, `js:/abs/path`, `node:/abs/path`, and `python:/abs/path` set the execution directory for `/run`, inline pipelines, and `/exe` definitions; execution fails on relative, missing, or non-Unix paths
- **Template collection imports**: Load entire directories of templates with shared parameter signatures
  - `/import templates from "@base/agents" as @agents(message, context)` imports all `.att`/`.mtt` files
  - Access by filename: `@agents["alice"](@msg, @ctx)` or nested: `@agents.support["helper"]`
- **Directory module imports**: Import a directory that loads each immediate subdirectory `index.mld`
  - Returns an object keyed by sanitized directory name with each `index.mld` export set as the value
  - Skips `_*` and `.*` directories by default, override with `with { skipDirs: [] }`
- **@exists() builtin**: Returns true when an expression evaluates without error (string args check paths; glob args require at least one match)
- **When separators**: Semicolons separate when arms in directives and expressions
- **Bound-value when-expressions**: `when @value first [...]` and `when @value [...]` support pattern arms like `>= 0.7`, `>= 0.3 && < 0.7`, and `*`
- **Nullish coalescing operator**: `??` returns the left operand when it is not nullish, otherwise returns the right operand
- **For-when filter sugar**: `for ... when ...` drops non-matches without null placeholders via implicit `none => skip` branch
- **Conditional inclusion (`@var?`)**: Universal pattern for conditionally including content based on truthiness
  - Commands/Templates: `@var?`...`` - include backtick template if truthy (e.g., `cmd { echo @tools?`--tools` }`)
  - Strings: `@var?"..."` - include quoted fragment if truthy (e.g., `"Hello @title?"@title "`)
  - Arrays: `[@a, @b?, @c]` - omit element if falsy
  - Objects: `{key?: @val}` - omit pair if value is falsy
  - Field access supported: `@obj.field?`...`` evaluates full path before truthiness check
- **Object mx helpers**: `@obj.mx.keys`, `@obj.mx.values`, and `@obj.mx.entries` expose object utilities
- **For-loop item keys**: `@item.mx.key` exposes the current object key (alongside `@item_key`)
- **Parallel execution error accumulation**: Errors accumulate in `@mx.errors` array instead of failing fast
  - `/for parallel @x in @xs [complex-block]` supported with block-scoped `let` only
  - Failed iterations produce error markers `{ index, key?, message, error, value }` in results
  - Parallel pipeline groups (`|| @a || @b || @c`) also accumulate errors
- **Comments in block bodies**: `>>` (start-of-line) and `<<` (end-of-line) comments work inside `[...]` blocks

### Changed
- **Renamed `@ctx` to `@mx`**: The execution context variable is now `@mx` ("mlld execution"). Access retry count via `@mx.try`, hints via `@mx.hint`, stage info via `@mx.stage`, etc. The `.ctx` metadata namespace on variables is now `.mx` ("metadata")—use `@file.mx.filename`, `@file.mx.tokens`, etc.
- `/exe` RHS pipe sugar accepts direct `@value | cmd { ... }` pipelines (legacy `run` form still works); identity definitions keep with-clause pipelines when evaluating parameters
- **Mode-aware parsing**: Environment variable `MLLD_STRICT=1` forces strict mode, `MLLD_STRICT=0` forces markdown mode, overriding file extension inference
  - FormatAdapterSink and TerminalSink are mutually exclusive
- **Import from @payload and @state**: Route files can now import fields from execute() payload and state
  - `/import { @message, @userId } from @payload` imports specific fields
  - `/import { @conversationId } from @state` imports state fields
  - Enables explicit, auditable access to runtime-injected data
  - Similar pattern to `/import { USER } from @input` for environment variables
- **Live @state and literal payload strings**: `@state` reads stay fresh after state writes, and `@payload/@state` dynamic modules emit literal strings so @mentions and user data do not interpolate.
- **LoadContentResult implements StructuredValue surface**: File loading now returns values with `.text`, `.data`, and `.mx` surfaces. Access metadata via `.mx.filename`, `.mx.tokens`, `.mx.fm` etc. Direct property access (`.content`, `.filename`) remains for backward compatibility but `.mx` is recommended for new code.
- **Simplified .keep usage**: Metadata now accessible in mlld contexts without `.keep` (e.g., `@file.mx.filename` works directly). `.keep` only needed when passing to JS/Node to preserve StructuredValue wrapper. Apply `.keep` at call site (`@process(@file.keep)`) rather than variable creation.

### Fixed
- **@debug variable stack overflow**: Fixed infinite recursion when accessing `@debug`. Variable metadata getter was calling itself recursively when building context snapshot.
- **Exe block return-only syntax**: Exe blocks can now return directly without a let statement: `exe @f() = [ => { name: "hello" } ]`
- **Method calls in object literal values**: Method calls like `@x.trim()` now work as object literal values in returns: `=> { file: @f.mx.relative, review: @review.trim() }`
- **`.mx` access in for loops**: File metadata (`@f.mx.relative`, `@f.mx.filename`) now accessible when iterating over glob results in all contexts: direct interpolation, object literals (`{ file: @f.mx.relative }`), and exe function parameters.
- **@json error clarity**: `@json` throws clear errors when parsing fails instead of silently mangling input. Detects markdown code fences and suggests `@json.llm`.
- **Pipeline filter error**: Writing `| json` instead of `| @json` now gives helpful error: "Pipeline filters require the @ prefix"
- **Arithmetic operators in exe blocks**: Math operators (`+`, `-`, `*`, `/`, `%`) work in exe blocks, let assignments, and return values
- **Universal StructuredValue model**: All runtime values flow as StructuredValues with `.text`, `.data`, and `.mx` surfaces. Boundaries use `asData()`/`asText()` for extraction. Fixes when-expressions returning numbers, object serialization, and numeric comparisons.
- **Field access precedence**: User data properties take precedence over guard quantifiers (`.all`, `.any`, `.none`). Core metadata (`.type`, `.mx`) always from Variable.
- **Standalone @ in double-quoted strings**: `@` not followed by identifier treated as literal character (`.startsWith("@")` now works)
- **Setup in nested directories**: `mlld setup` detects parent config and prompts to update parent or create new local config
- Effect actions (`show`, `log`, `output`, `append`) work uniformly in all RHS contexts
- Streaming no longer produces duplicate output when using format adapters
- Regex arguments are parsed as RegExp values, so `.match(/.../)` conditions (including grouped patterns) work in when-expressions and other exec calls without falling back to strings
- Block directive parse errors reparse with correct offsets for better error locations
- Registry publish flow improvements: recreates missing fork refs, minimal PR bodies, better error messages
- Module installer honors requested versions by purging mismatched cache entries
- Lock file normalization strips version suffixes to prevent duplicates
- Variable boundary escaping (`@var\.ext`) works in all interpolation contexts
- `@@` and `\@` both escape to literal `@`
- Template paths support `@var` interpolation in double-quoted strings
- CLI `--payload` alias for `--inject`
- **ESM bundle compatibility**: MJS bundle fixed for Node 24+ ESM projects (converted `require()` calls to ESM imports)
- **LSP: Mode-aware highlighting**: Language server detects `.mld` (strict) vs `.mld.md` (markdown) and highlights bare directives correctly; text content in strict mode shows diagnostics; completions adapt to mode
- **/var augmented assignment errors**: Invalid `@x = @y += @z` errors stop at the `+=` instead of earlier lines in LSP diagnostics
- **Exe block += evaluation**: `let @result = @a; @result += @b` concatenates arrays instead of replacing them
- `run` statements work inside `[...]` blocks for `/exe` and `/for` bodies
- Fixed `/run @value | cmd { ... }` parsing so `@value` becomes `with { stdin: ... }` (matches `/exe` RHS pipe sugar)
- **LSP: rc78 syntax support**: Semantic tokens for block syntax `[...]`, `let` keyword, `+=` augmented assignment, `while`/`done`/`continue`, `stream` directive, working directories `cmd:/path`, when semicolons
- **LSP bug fixes**: When block assignments ([#327](https://github.com/mlld-lang/mlld/issues/327)), pipe transform parity ([#328](https://github.com/mlld-lang/mlld/issues/328)), EOL comments in when ([#329](https://github.com/mlld-lang/mlld/issues/329)), variable interpolation in /run ([#330](https://github.com/mlld-lang/mlld/issues/330)), function execution in /run ([#331](https://github.com/mlld-lang/mlld/issues/331)), array/object value highlighting ([#332](https://github.com/mlld-lang/mlld/issues/332))
- **LSP debugging tools**: `npm run validate:tokens`, `npm run test:nvim-lsp <file>` for testing semantic highlighting
- **LSP tokenization fixes**: Negative char positions causing Neovim crashes, missing visitor registrations (field/numericField/arrayIndex/LetAssignment/ExeReturn), container object recursion in visitChildren(), ExecInvocation wrong token type.

## [2.0.0-rc77]

### Added
- **CLI `--structured` mode**: New `--structured` flag outputs JSON with effects, exports, stateWrites, and full security metadata for auditing and programmatic consumption
- **CLI `--inject` flag**: Runtime module injection via `--inject @module=value` or `--inject @module=@file.json`. Enables testing with mock data and dynamic context without temp files. Multiple `--inject` flags supported.
- **MCP static analysis**: `mlld mcp` now uses `analyzeModule()` for tool discovery instead of code execution, improving security by discovering tools without running arbitrary code
- **SDK execution modes**: `interpret(mode)` with four modes for different consumption patterns
  - `document` (default): Returns plain string output
  - `structured`: Returns `{ output, effects, exports, environment }` with security metadata on all effects
  - `stream`: Returns `StreamExecution` handle with real-time event delivery (`.on()`, `.off()`, `.done()`, `.result()`, `.abort()`)
  - `debug`: Returns `DebugResult` with AST, variables, ordered trace, and timing
- **Dynamic module injection**: `processMlld(script, { dynamicModules: {...} })` enables runtime context injection without filesystem I/O. All dynamic imports automatically labeled `src:dynamic` for guard enforcement. Enables multi-tenant applications (inject per-user/project context from database). Optional `dynamicModuleSource` parameter adds custom source labels (e.g., `src:user-upload`, `src:database`) for fine-grained guard policies distinguishing between trusted and untrusted dynamic data.
- **State write protocol**: `/output @value to "state://path"` captures state updates as structured data instead of writing to filesystem. State writes included in `StructuredResult.stateWrites` with security metadata.
- **SDK runtime execution**: `execute(filepath, payload, options)` provides file-based route execution with in-memory AST caching, state hydration (`@state`, `@payload`), timeout/cancellation, and full effects logging.
- **SDK analysis tools**: `analyzeModule(filepath)` extracts capabilities, imports, exports, guards, and security metadata without execution. Enables static analysis, capability checking, and module introspection.
- **Effect security metadata**: All effects in structured/stream/debug modes include `security` field with labels, taint tracking, and provenance for auditing and policy enforcement.
- **Execution events**: `ExecutionEmitter` bridges streaming infrastructure to SDK events (`stream:chunk`, `command:start/complete`, `effect`, `execution:complete`) for real-time monitoring.
- **Directory-based taint tracking**: File loads now include `dir:*` labels for all parent directories, enabling guards like `@input.ctx.taint.includes('dir:/tmp/uploads')` to prevent executing uploaded files.

### Changed
- **Security model streamlined**: `SecurityDescriptor` now uses `taint: DataLabel[]` (accumulated labels) instead of single `taintLevel` enum. Automatic labels added: `src:exec` (commands), `src:file` (file loads), `src:dynamic` (runtime injection), `dir:/path` (file directories).
- Effect handler now records effects when `mode: 'structured' | 'stream' | 'debug'`; default `document` mode skips recording for performance.
- **`mlld run` now uses `execute()`**: Run command leverages AST caching, metrics, and timeout support from SDK's `execute()`. New `--timeout` and `--debug` flags available.

### Fixed
- **Whitespace normalization** ([#396](https://github.com/mlld-lang/mlld/issues/396)): Introduced OutputIntent abstraction with collapsible breaks to eliminate extra blank lines. Newlines from document structure now collapse automatically, producing consistent output spacing.
- **Prettier dependency removed** ([#281](https://github.com/mlld-lang/mlld/issues/281)): Replaced Prettier with simple line-based normalizer. Eliminates hanging bug, removes JSON protection hacks, and improves performance (~0ms vs ~50ms). The `@md` transformer now normalizes output (strips trailing whitespace, collapses blank lines) rather than reformatting.
- Array slicing now supports variable interpolation in slice indices ([#457](https://github.com/mlld-lang/mlld/issues/457)). Previously `@arr[0:@limit]` would fail to parse; now `@arr[@start:@end]`, `@arr[0:@limit]`, and `@arr[@offset:]` all work as expected.
- Fixed issue where `/var @item = cmd {..}` would fail due to missing grammar pattern
- Pipeline effects (`output`, `show`, `append`, `log`) run through guard pre/post hooks. `op:output`/`op:show`/`op:append`/`op:log` guards block both directives and inline effects; guard retries on effects deny with a clear message.

## [2.0.0-rc76]

### Fixed
- Circular reference detection for executables ([#255](https://github.com/mlld-lang/mlld/issues/255)): mlld now detects when an executable calls itself recursively without a terminating condition and throws a clear `CircularReferenceError` instead of causing a stack overflow. This includes both direct recursion (`@f()` calling `@f()`) and mutual recursion (`@ping()` ↔ `@pong()`). Legitimate patterns like pipeline retries and builtin method calls are excluded from detection.
- Liberal import syntax: quoted module paths like `"@local/module"` and `"@base/file.mld"` now work alongside unquoted forms ([#300](https://github.com/mlld-lang/mlld/issues/300)). The interpreter detects resolver patterns in quoted strings and routes them correctly instead of treating them as variable interpolation.
- ProjectPathResolver now recognizes `.mld.md` and `.mlld.md` extensions as modules, fixing imports from `@base/...` paths
- SpecialVariablePath in grammar now stops at line boundaries, preventing path parsing from consuming content across newlines

## [2.0.0-rc75]

### Added
- Object spread syntax `{ ...@var, key: value }` for composing objects with left-to-right overrides; spreading non-objects now errors.
- Augmented assignment `@var += value` for local variable accumulation in when blocks. Supports arrays (concat), strings (append), and objects (shallow merge). Only works with local `let` bindings, maintaining global immutability.

### Fixed
- Improved error message for alligator field access inside XML/HTML tags - now detects pattern and suggests variable workaround
- Fixed `as` transform pattern for glob patterns - `<*.md> as "### <>.ctx.filename"` now correctly transforms each file instead of returning empty array
- `/show` directive now errors on multiple arguments instead of silently ignoring extras ([#370](https://github.com/mlld-lang/mlld/issues/370)). Use templates for multiple values: `/show \`@a @b\``

## [2.0.0-rc74]

### Fixed
- `run cmd {...}` syntax now works consistently in `/var` and `/exe` contexts, not just `/run` directives. Previously `/var @x = run cmd {echo "hi"}` and `/exe @f() = run cmd {echo "hi"}` would fail to parse. Both `run {...}` (implicit cmd) and `run cmd {...}` (explicit) are now supported everywhere for backwards compatibility.

## [2.0.0-rc73]

### Added
- `let` keyword for local variables in `/when` blocks: `let @x = value` creates block-scoped variables before conditions, enabling cleaner conditional logic without polluting outer scope
- `/run cmd {command}` syntax for shell commands, consistent with `cmd {..}` in other contexts. Bare `/run {command}` still works for backwards compatibility.
- AST selector wildcards ([#505](https://github.com/mlld-lang/mlld/issues/505)): `{ handle* }`, `{ *Validator }`, `{ *Request* }`, `{ get? }` for pattern-based symbol matching
- AST type filters: `{ *fn }`, `{ *var }`, `{ *class }`, `{ *interface }`, `{ *type }`, `{ *enum }`, `{ *struct }`, `{ *trait }`, `{ *module }`, `{ * }` to get all definitions of a specific type
- AST name listing: `{ ?? }`, `{ fn?? }`, `{ var?? }`, `{ class?? }` return string arrays of definition names instead of code
  - Single file: returns plain string array for simple iteration
  - Glob patterns: returns per-file structured results `[{ names: string[], file, relative, absolute }]` enabling queries like `/for @f in <**/*.py { class?? }> => show "@f.names.length classes in @f.relative"`
- Section listing for markdown: `# ??`, `# ##??`, `# ###??` return arrays of heading titles
  - Single file: plain string array
  - Glob patterns: per-file structured results `[{ names: string[], file, relative, absolute }]`
- Variable interpolation in AST selectors: `{ *@type }`, `{ @type?? }` for dynamic pattern construction
- Usage patterns with wildcards and type filters: `{ (handle*) }`, `{ (*fn) }` find functions that use matched symbols
- Validation: mixing content selectors with name-list selectors now throws clear error
- LSP/syntax highlighting: `/guard`, `/stream`, `/append`, `/export` directives; guard keywords (`before`, `after`, `always`, `allow`, `deny`, `retry`); `let`/`var` in when blocks; import types (`module`, `static`, `live`, `cached`, `local`); data labels; pipeline operators (`|`, `||`); type-checking methods (`.isArray()`, etc.); AST selector patterns

### Changed
- **BREAKING**: Variable assignments in `/when` actions now require explicit `var` prefix. Use `var @x = value` for outer-scope variables, `let @x = value` for block-local variables. Bare `@x = value` syntax now throws an educational error.

### Fixed
- Field access with pipes in `/show` now correctly extracts field values before piping ([#506](https://github.com/mlld-lang/mlld/issues/506)). Previously `@data.0.code | cmd {head -3}` would pipe the parent array instead of the code field value. Field access now happens before pipeline processing for both `VariableReference` and `VariableReferenceWithTail` node types.
- Export directive grammar now correctly distinguishes guards from variables ([#498](https://github.com/mlld-lang/mlld/issues/498)). Previously all exports were marked as `guardExport`, breaking `/export` for executables and variables. Now uses runtime guard registry check.
- `/export` directive now recognized by grammar context detection - added missing `export` keyword to `DirectiveKind` enum. Export filtering now works correctly for namespace imports.
- `/export { * }` wildcard syntax now parses correctly - added `*` as valid export identifier
- Module tests updated to use current `/export { name }` and `/exe @func() = \`...\`` syntax
- Documentation updated: `/export guard @name` changed to `/export { @guardName }`
- JSON field access in executables now requires explicit `.data` accessor (e.g., `@var.data.field`)
- Glob pattern test files renamed with unique prefixes to prevent virtual filesystem collisions
- Frontmatter access in glob results now uses `.ctx.fm.field` accessor
- Test expectations updated for current JSON formatting and blank line behavior

## [2.0.0-rc72]

### Added
- Type-checking builtin methods: `.isArray()`, `.isObject()`, `.isString()`, `.isNumber()`, `.isBoolean()`, `.isNull()`, `.isDefined()` return booleans for conditional logic ([#414](https://github.com/mlld-lang/mlld/issues/414)). Note: `.isDefined()` safely returns `false` for missing variables or fields without throwing.

### Fixed
- Method chaining after array access now works: `@msg.split("_")[0].toUpperCase()` ([#408](https://github.com/mlld-lang/mlld/issues/408))
- `mlld init` is now path-aware - selecting "llm/modules" while already in that directory no longer creates nested paths ([#453](https://github.com/mlld-lang/mlld/issues/453))

### Removed
- Grammar cleanup: Removed undocumented `when any` and `when all` modifiers. Use `&&` and `||` operators for AND/OR logic in conditions.

## [2.0.0-rc71]

### Fixed
- JSON escape sequences (`\n`, `\t`, etc.) now preserved when piping data through shell commands like `echo` ([#456](https://github.com/mlld-lang/mlld/issues/456)). Previously, escape sequence normalization in command executables would convert properly-escaped `\\n` back to actual newlines, corrupting JSON data.
- Pipeline synthetic source stage preserves StructuredValue wrappers so with-clause pipelines keep JSON arrays/objects intact through `/exe` when-expression actions ([#461](https://github.com/mlld-lang/mlld/issues/461)).

## [2.0.0-rc70]

### Added
- Streaming support: `stream` keyword, `/stream` directive, and `with { stream: true }` enable live chunk emission with progress sinks and executor streaming (shell, bash, node). Parallel groups stream concurrently and buffer results. Suppress with `--no-stream` or `MLLD_NO_STREAM`.
- Streaming UX MVP:
  - Auto-parse NDJSON for `stream` execs (paths: message.content[].text/result/delta.text/completion/error.message).
  - Live stdout for message text with spacing/dedupe; thinking/tool-use to stderr (`💭 text`, `🔧 name input=preview`); tool results suppressed for noise.
  - Raw event visibility: `--show-json` (or `MLLD_SHOW_JSON=true`) mirrors NDJSON to stderr; `--append-json [file]` writes NDJSON to JSONL (default `YYYY-MM-DD-HH-MM-SS-stream.jsonl` when omitted).
  - Streaming `/show ...` avoids double-print of streamed content.

### Security

**Guards**:
- Policy enforcement for data access and operations
- Syntax: `/guard <label> { allow/deny/retry }` with optional conditions
- Guards trigger on data labels (`secret`, `pii`) or operations (`op:run`, `op:exe`)
- Support `allow`, `deny`, and `retry` decisions
- Can transform data with `allow @transform(@input)`
- Fire before operations (input validation) or after operations (output validation)

**Expression Tracking**:
- Guards see security labels through all transformations (closes `@secret.trim()` bypass hole)
- Provenance preserved through: chained builtin methods, template interpolation, field access, iterators, pipelines, nested expressions
- Example: `@secret.trim().slice(0, 5)` preserves `secret` label through entire chain
- Guards fire at directive boundaries, exe invocations, and pipeline stages

**Guard Composition**:
- All guards execute in registration order (file top-to-bottom, imports flatten at position)
- Multiple guards compose with decision precedence: deny > retry > allow @value > allow
- Transform chaining: Guard N output → Guard N+1 input with full provenance tracking
- Guard history exposed via `@ctx.guard.trace/hints/reasons` for denied handlers
- Pipeline guard history via `@p.guards` tracks guard activity across all pipeline stages
- Deterministic IDs for unnamed guards: `<unnamed-guard-N>`
- Per-input guards (data labels) and per-operation guards (`op:run`, `op:exe`, etc.)

**Before/After Guards**:
- Guards fire before operations (input validation) or after operations (output validation)
- Syntax: `/guard @name before datalabel = when [...]` where TIMING is `before`, `after`, or `always`
- Syntactic sugar: `/guard @name for LABEL` is equivalent to `before` timing (explicit `before` recommended)
- Context: `@input` in before guards, `@output` in after guards, both available in denied handlers
- Execution order: before guards → operation → after guards
- Retries supported in pipeline context for both before and after guards

**Allow @value Transforms**:
- Guards transform inputs/outputs: `allow @redact(@input)` or `allow @sanitize(@output)`
- Transforms chain with metadata preservation (`guard:@name` appended to sources)
- Works in both before guards (input sanitization) and after guards (output sanitization)
- Cross-scope chaining: per-input guard transforms flow to per-operation guards
- Provenance tracking: labels union, taint maximum, sources accumulate

**Guard Overrides**:
- Per-operation control: `with { guards: { only: [...], except: [...], false } }`
- `guards: false` disables all guards (emits warning to stderr)
- `only: ["@guard"]` runs specified guards only (unnamed guards excluded)
- `except: ["@guard"]` skips named guards (unnamed guards still run)
- Conflict detection: throws error if both `only` and `except` specified

### Added
- StructuredValue `.ctx`/`.internal` surfaces power provenance, security, and behavior metadata
- `/append` directive and `| append` pipeline builtin for incremental file writes (JSONL/text) with shared `/output` source evaluation
- `@json.llm` transformer extracts JSON from LLM responses with code fences or embedded prose. Returns `false` when no JSON found.
- `@json.fromlist` transformer converts plain text lists (one item per line) to JSON arrays
- Chained builtin methods on variables: string methods (slice, substring, substr, replace, replaceAll, padStart, padEnd, repeat, split, join) and array methods (slice, concat, reverse, sort) work in chains like `@secret.trim().slice(0, 6)` with security labels preserved
- Structured-value helpers: added `keepStructured`/`keep` helper and `.keepStructured`/`.keep` field-access sugar to retain metadata/provenance without unwrapping content. Built-in `@keep`/`@keepStructured` executables allow helper-style usage in scripts.
- For loops accept dotted iteration variables and bind both the base element and its field (e.g., `for @item.path in @files`) with proper field access errors.
- For loop bodies can be `when [...]` across /for, /var, and /exe, using first-match semantics per iteration and feeding branch results into loop outputs.

- Alligator JSON ergonomics: `<*.json>` and `<*.jsonl>` auto-parse to StructuredValues (parsed `.data`, raw `.text`, `.ctx` preserved); use `.text` when raw strings are needed.

### Fixed
- Templates now correctly parse comparison operators like `<70%` and `< 70` instead of treating them as file references

- Inline `/for` loops in templates only trigger at line start (not mid-line)
- **When-expression pipelines**: `/exe … = when [...]` actions now accept `| append`, `| log`, `| output`, and `| show` stages without misparsing ternary expressions (fixes `slash/when/exe-when-expressions-operators`).

- Backtick and `::` templates handle XML-like tags identically

- Fixed false circular reference warnings when parallel tasks load the same file
- Inline pipeline effect detection now differentiates builtin `append` from user-defined commands, restoring stage execution for execs named `append`
- Alligator syntax in for expressions: `for @f in @files => <@f>` and property access like `for @f in @files => <@f>.fm.title` now work correctly
- Module content suppression during imports - imported module content no longer appears in stdout
- Shell pipe detection respects quoted strings - pipe characters inside quoted arguments no longer trigger pipe handling
- Transformer variant resolution in pipelines - `@json.fromlist`, `@json.llm`, `@json.loose`, and `@json.strict` work correctly in all pipeline contexts
- Alligator `.relative` resolves from inferred `@base` (or the script path when base is unavailable) so metadata matches project-root paths
- Comma in `when` condition lists now emits a targeted parse error instead of a generic /exe syntax failure
- Wrong parallel syntax order (`/for parallel 18` instead of `/for 18 parallel`) now shows helpful error with correct syntax examples

### Changed
- Braced commands require explicit `cmd { ... }`; bare `{ ... }` parses as structured data, pipelines accept inline value stages with structured output, and bare brace commands raise a targeted parse error
- Enhanced error message for `run sh` in `/exe` explains distinction between bare commands and shell scripts
- Shell commands now run from project root when `@base` is inferred, otherwise from script directory
- `/for` parallel syntax uses `parallel(cap, pacing)` instead of `(cap, pacing) parallel`. Old syntax still parses with a warning.
- Unified file loading uses StructuredValue metadata consistently: text files unwrap to strings by default, JSON/JSONL unwrap to parsed objects/arrays, `.ctx` carries file/URL metadata, `.keep` passes wrappers into JS/Node, and `MLLD_LOAD_JSON_RAW` is removed in favor of `.text` for raw access.

## [2.0.0-rc69]
### Fixed
- JS and Node executors treat expression-style blocks as implicit returns, so `/var` assignments and pipelines receive native objects/arrays and property access like `@repo.name` works without helper wrappers.
- Node shadow executor surfaces the underlying runtime error message while still cleaning up timers, restoring the `node-shadow-cleanup` regression coverage.

## [2.0.0-rc68]
### Fixed
- Template executables detect JSON-looking strings and wrap them as structured values, so downstream pipelines receive native objects instead of escaped text (#435).
- Foreach iteration normalizes stage outputs that stringify JSON and passes parsed arrays/objects forward, restoring the behaviour users expect from `| @json` inputs.
- `/show` array rendering unwraps structured elements to their `.data` view when possible, keeping canonical text intact for load-content metadata and structured JSON displays.

## [2.0.0-rc67]
### Fixed
- Pipelines sanitize JSON-like shell output by escaping control characters inside string literals, so `/run` stages that echo newline-bearing JSON feed structured data forward correctly.

## [2.0.0-rc64]
### Fixed
- **Alligator section parsing with "as" substring**: Fixed grammar bug where section names containing "as" (like "Gotchas", "Installation", "Basic Usage") were rejected
  - Changed `AlligatorSectionChar` rule from `!"as"` to `!(" as")` to only exclude the `as` keyword when used for renaming
  - Section syntax like `<file.md # Gotchas>` now works correctly
  - Distinguishes between section names with "as" and the rename keyword: `<file.md # Section> as "New Title"`
  - Added test coverage in `tests/cases/slash/show/alligator-section-as-substring/`
- Preserved structured pipeline outputs across chained executables by wrapping JSON-like strings returned from JS/Node stages, preventing downstream stages from receiving `[object Object]` text (#435).
- Updated run/exec structured handling and regression fixtures so batch/parallel pipelines, foreach separators, and retry fallbacks assert native arrays/objects instead of stringified JSON, closing the remaining gaps from #435 user scenarios.

## [2.0.0-rc63]
### Fixed
- Fixed local resolver to recognize all mlld extensions as modules when they contain directives. Previously only .mlld.md files were explicitly treated as modules, causing "Import target is not a module" errors when importing .mld files via custom resolver prefixes like @context/.
- Improved content type detection to parse file contents for mlld directives across all module extensions, maintaining backward compatibility for files with non-standard extensions that contain valid mlld code.
- Missing `--tag` on cli added
- Update docs to cover modules, registry, resolvers

### Added
- Batch and condensed pipeline stages now receive the structured wrapper instead of raw strings, so helpers can work with native arrays/objects without JSON.parse.
- **Custom tag support for publishing**: `mlld publish --tag <name>` allows publishing modules with custom version tags
  - Publish with beta/alpha tags: `mlld publish module.mld --tag beta`
  - Tag validation ensures alphanumeric + hyphens only, 2-50 character length
  - Reserved tags (`latest`, `stable`) are rejected with clear error messages
  - Users can import using custom tags: `/import { @helper } from @alice/utils@beta`

## [2.0.0-rc62]

### Added
- **Batch pipelines for collection expressions**: `for` and `foreach` now accept a trailing `=> |` pipeline that runs after iteration completes. The batch phase reuses standard pipeline syntax, applies to the gathered array, and may return arrays, scalars, or objects. Grammar attaches the pipeline to `ForExpression.meta.batchPipeline` and `ForeachCommandExpression`, and the interpreter processes the results via `processPipeline()` before emitting the final variable or display output.

### Notes
- Batch pipelines behave like condensed pipelines: each stage receives string input, so helpers that expect arrays should parse the string back to JSON. Currently parallel groups (`||`) share the same semantics but are not fully supported/tested.

## [2.0.0-rc61]

### Added
- **Loose JSON parsing modes**: `@json` now accepts relaxed JSON syntax (single quotes, trailing commas, comments) using JSON5, with explicit `@json.loose` and `@json.strict` variants for opting in or enforcing strict parsing. Error messages direct users to the loose mode when strict parsing fails.

### Fixed
- **Structured data handling in field access**: Fixed array operations on nested StructuredValue wrappers
  - Field access now properly unwraps nested StructuredValue before array operations
  - Fixes potential runtime errors with deeply nested structured data (e.g., `@nested[0]` where `@nested` is a wrapped array)
  - Related to #435 structured data edge cases
  - Fixed in `interpreter/utils/field-access.ts:477` and `:248`

- **Exec invocation stdin handling**: Fixed stdin coercion missing StructuredValue unwrapping
  - Exec invocations now properly unwrap StructuredValue when preparing stdin data
  - Aligns with run.ts stdin handling (same pattern as the golden standard)
  - Prevents double-wrapping or incorrect stringification of structured values passed via stdin
  - Related to #435 structured data edge cases
  - Fixed in `interpreter/eval/exec-invocation.ts:49`

- **Shell interpolation of structured values**: Complex arrays/objects now survive shell argument quoting
  - Shared `classifyShellValue` helper drives `/run` and `@exe` stdin/argument coercion
  - Interpolation tracks both single- and double-quoted spans, avoiding `[object Object]` and broken quoting
  - File-content fixtures confirm literal `$`, `` ` ``, and quotes reach the shell intact
  - Covers regressions from #435 user scenario

- **Variable assignment with wrapped values**: Fixed String() conversions producing [object Object]
  - Variable assignments now use `valueToString()` helper that checks for StructuredValue wrappers
  - Uses `asText()` helper for StructuredValue wrappers instead of naive String() conversion
  - Applies fix to 7 locations in var.ts where String() was used on complex values (lines 725, 751, 763, 773, 782, 820, 823)
  - Variable type detection now properly unwraps StructuredValue before Array.isArray() checks (3 locations: lines 719, 745, 757)
  - Related to #435 structured data edge cases
  - Fixed in `interpreter/eval/var.ts`

## [2.0.0-rc60]

### Fixed
- **Shell command interpolation with nested arrays**: Fixed arrays of objects/arrays being converted to `[object Object]` in shell commands
  - Shell command context (e.g., `echo @array`) now properly JSON-stringifies complex array elements
  - Previously `String(object)` produced `[object Object]`, breaking data flow through shell executables
  - Example: `/exe @func(e) = run { echo @e }` now correctly outputs JSON for nested arrays
  - Fixes remaining edge case from #435 (https://github.com/mlld-lang/mlld/issues/435#issuecomment-3386904732)
- Addressed instances of old mlld.lock.json file expectations throughout codebase

## [2.0.0-rc59]

### Changed
- **CLI commands aligned with new config file naming**: Updated all CLI commands to reference the new dual-file configuration system
  - `mlld-config.json`: User-editable project settings (dependencies, preferences, resolver configuration)
  - `mlld-lock.json`: Auto-generated lock file (versions, hashes, sources)
  - Replaced `mlld.lock.json` references throughout CLI commands and help text
  - Commands updated: `setup`, `alias`, `run`, `init-module`
  - Editor integrations updated: Neovim LSP, VS Code Language Server
  - Backward compatibility maintained: LSP and editor tooling check for old `mlld.lock.json` as fallback
  - All commands now use `ProjectConfig` abstraction

## [2.0.0-rc58]

### Fixed
- **Foreach with structured values**: `foreach` now unwraps StructuredValue arguments
  - Previously failed with "got structured text" when array came from pipeline
  - Example: `/var @chunked = @data | @chunk(2)` then `foreach @process(@chunked)` now works
  - Aligns with JavaScript stages which already unwrap automatically

## [2.0.0-rc57]

### Added
- **MCP server**: `mlld mcp` serves exported `/exe` functions as MCP tools
  - Exposes functions over JSON-RPC stdio transport
  - Default discovery: `llm/mcp/` directory when no path specified
  - Config modules: `--config module.mld.md` exports `@config = { tools?, env? }`
  - Environment overrides: `--env KEY=VAL` (MLLD_ prefix required)
  - Tool filtering: `--tools tool1,tool2` or via config
  - Duplicate tool names halt with error showing conflicting sources
  - Example: `/exe @greet(name) = js { return \`Hello ${name}\`; }` becomes `greet` tool

### Changed
- **Data flow between stages**: Native types preserved throughout pipelines
  - Loaders return parsed data: `<data.json>` yields object, not JSON string
  - Pipeline stages pass arrays/objects directly: `@data | @process` receives native type
  - JavaScript functions receive parsed values without `JSON.parse()`
  - Templates and output convert to text automatically
  - Fixes #435 

### Breaking
- Remove `JSON.parse()` calls in JavaScript stages - will fail on already-parsed data
- Use `.text` to access stringified data, `.data` to get structured data in string context 
- Pipelines expecting JSON strings will receive objects/arrays instead

## [2.0.0-rc56]

### Added
- **Import Types System**: Control how modules and resources are resolved
  - `module` imports: Pre-installed registry modules (offline after install)
  - `static` imports: Content embedded at parse time (zero runtime cost)
  - `live` imports: Always fresh data (fetched every execution)
  - `cached(TTL)` imports: Smart caching with time limits (5m, 1h, 7d, etc.)
  - `local` imports: Direct access to development modules in `llm/modules/`
  - Example: `/import module { api } from @corp/tools`, `/import cached(1h) <https://api.example.com> as @data`

- **Module management**:
  - `mlld install @author/module`: Install modules from public registry
  - `mlld update`: Update modules to latest compatible versions
  - `mlld outdated`: Check for available updates
  - `mlld ls`: View installed modules with status and sizes
  - Registry integration with CDN-distributed module catalog

- **Configuration Files**:
  - `mlld-config.json`: Your project settings (dependencies, preferences)
  - `mlld-lock.json`: Auto-generated locks (versions, hashes, sources)

- **Simplified Development Workflow**:
  - Use `/import local { helper } from @author/module` to access modules in `llm/modules/` using published name (if you are @author or can publish to private @author registry)
  - Useful for iterating on modules before publishing

### Changed
- Import syntax now requires `@` prefix on imported names: `/import { @helper } from module`
- Module publishing requires explicit `/export { ... }` manifests
- Import failures now stop execution (exit code 1) instead of continuing
- Smart import type inference based on source patterns
- Pipelines support leading `||` operator for immediate parallel execution: `/var @result = || @a() || @b() || @c()` runs all three functions concurrently
- Leading parallel syntax works in `/var`, `/run`, and `/exe` definitions
- Pipeline concurrency controls: `(n, wait)` shorthand syntax and `with { parallel: n, delay: wait }` for caps and pacing

### Fixed
- Module installation fetches from real registry instead of placeholders
- Version resolution respects "latest" tags and semantic versioning
- Module integrity verified with SHA-256 hashes

## [2.0.0-rc55]

### Added
- Stdin support for `/run` directive and `/exe` definitions:
  - New syntax: `/run { command } with { stdin: @variable }` passes data directly via stdin without shell escaping
  - Pipe sugar: `/run @data | { command }` normalizes to `with { stdin: @data }` for cleaner syntax
  - Works in executable definitions: `/exe @func(data) = run { command } with { stdin: @data }`
  - Pipe sugar in executables: `/exe @func(data) = run @data | { command }`
  - Eliminates JSON double-stringification when passing structured data to commands like `jq`, `cat`, etc.
  - Preserves shell safety while enabling proper JSON/CSV/XML data flow through pipelines

- JSON data access pattern for JavaScript functions (addresses #428):
  - `.data` and `.json` accessors parse JSON strings during variable evaluation before passing to functions
  - `.text` and `.content` accessors preserve original string content
  - Eliminates need for manual `JSON.parse()` calls in JavaScript functions
  - Works consistently across files, variables, and command output
  - Example: `/var @json = '{"items": []}'; /run @process(@json.data)` passes parsed array to function

- Native mlld functions in pipelines:
  - `/exe` functions using `for` and `foreach` constructs now work as pipeline sta