kestrel.markets/docs/adr/0030-the-grammar-is-measured-not-designed.md

A typed, token-efficient language + runtime for agentic trading: agents author bounded plans, the runtime fires them at the tick. CLI + typed library + MCP server.

# The grammar is measured, not designed — authoring-error clusters drive prompt and syntax evolution

**Status:** Accepted (owner, 2026-07-13) — the principle behind the repair-retry half of the ADR-0029
bounded authoring loop (epic kestrel-d72). Extends ADR-0009 (**the screen is measured, not
designed** — no rendering is blessed by taste; the converged screen is discovered under opportunity
cost) and ADR-0029 (**the agent-requested emergent View authoring loop** — the bounded, above-the-line
loop whose telemetry is the emergence study's raw data). This ADR is the language-design analog of
ADR-0009: where ADR-0009 says the *screen* is discovered from what agents look at, this says the
*grammar* is discovered from what capable agents author correctly, and the measurement instrument for
both is the same error-cluster telemetry the ADR-0029 loop already emits.

## Context

Kestrel's surface syntax (ADR-0001 — `View`/`Wake`/`Plan`/`Grade`; the multi-line, indentation-
structured document; the price-expression algebra) is today a *designed* artifact. It was authored by
hand, taught by worked example in the prompt profile (`authoring-v1`, `src/session/harness/prompt.ts`),
and defended by a fail-closed parser (`src/lang/parse.ts`). The parser is honest — it never silently
repairs — but honesty alone does not tell us whether the syntax is *good*, only whether a given string
conforms.

The dry-run-1 live baseline made the gap concrete and measurable. Three frontier models wasted 30–47%
of their turns on the **same** parse escape — *"unexpected `DO` after the WHEN clause"* — because they
collapsed the multi-line `PLAN … / WHEN … / DO …` document onto one line
(`docs/results/dry-run-1-live-baseline.md`; recorded in the `authoring-v1` profile's own header
comment). Two responses were possible. We took the first (teach the layout better in the prompt); the
`authoring-v1` profile is that fix. But the failure *cluster* — one error, dominating, shared across
independent capable models — is also a signal about the **syntax itself**: a grammar in which the
single most common mistake made by capable authors is a line-break convention is a grammar carrying
avoidable friction. The prompt fix and a hypothetical grammar relaxation (accept `WHEN … DO …` on one
line) are two readings of the *same measurement*. Nothing in the codebase yet treats that measurement
as a first-class, aggregated instrument rather than an anecdote a human noticed in one results file.

ADR-0029's repair-retry loop changes that. Every invalid author in the bounded OPEN loop now produces
a structured **parse-error / repair event** in the off-graded-path evidence log: the malformed input
signature, the repair-guiding error the parser returned, the model, and the repair outcome (did the
next attempt parse?). Aggregated across models and sessions, these events are exactly the corpus that
turns "a human noticed models collapse the PLAN block" into "across N models and M sessions, error
cluster C accounts for X% of first-attempt authoring failures, and Y% of repairs that follow error
message E succeed on the next attempt." That corpus is a measurement instrument for the language.

## Decision

**The grammar is measured, not designed. Aggregated authoring-error clusters — not taste — are the
evidence for evolving BOTH the prompt AND the syntax. Syntax evolves toward what capable agents author
correctly.**

Three commitments follow.

1. **The error-cluster telemetry is the measurement instrument.** The ADR-0029 repair loop's per-
   iteration evidence records (keyed by `(OPEN_ORDINAL, iteration)`, off the graded path — ADR-0029 §5)
   capture, for every invalid author: the **malformed input signature** (a normalized fingerprint of
   what the model wrote — not the raw bytes, which are strategy-bearing), the **repair-guiding error**
   the parser returned, the **model** identity, and the **repair outcome** (repaired-next-attempt /
   still-invalid / abandoned to budget-exhaustion). Aggregation over this log yields, per error
   cluster: its frequency, its cross-model breadth (one model's quirk vs a shared stumbling block),
   and — crucially — the **repair-success rate conditioned on the error message**. That last quantity
   grades the *error message* itself: a repair-guiding message after which capable models reliably fix
   their author on the next attempt is a good message; one they cannot recover from names a real
   grammar-ergonomics problem (or a bad message). This is the language analog of ADR-0009's "log every
   pane addition/removal and its stated reason."

2. **Clusters drive prompt refinement first (cheap, reversible, no ConfigId churn to the language).**
   The first response to a dominant, cross-model error cluster is to teach around it in a **new,
   versioned prompt profile** (as `authoring-v1` did for the WHEN/DO collapse) — never to silently
   patch the frozen one. A prompt change mints a new `promptHash` ⇒ a new `ConfigId` ⇒ a distinct grid
   column (ADR-0013), so the improvement is *measured* against the profile it replaces rather than
   assumed. A cluster that a prompt fix dissolves was a *teaching* gap, not a *grammar* gap — and the
   telemetry distinguishes the two: if the cluster survives a genuine, well-taught prompt profile
   across capable models, the friction is in the syntax, not the instruction.

3. **A cluster that survives good teaching is evidence for grammar/syntax evolution — under the same
   discipline as ADR-0009's screen freeze.** When an error cluster persists across a corrected prompt
   profile and multiple capable models, that is the signal to consider *changing the grammar* (relax a
   convention, accept a synonym, widen an accepting form) so the syntax bends toward what capable
   authors already write. Such a change is owner-gated and, like ADR-0009's discovered-View freeze, is
   **discovered then frozen**: proposed from the aggregate, adopted deliberately, and then held stable
   as a new language version — never auto-applied from a convergence statistic, and never a
   fail-*open* loosening that would let the parser silently accept what it once refused (fail-closed
   honesty, ADR-0004/0011, is non-negotiable — the parser stays exact; what evolves is *which exact
   grammar* it enforces).

Concretely, the measurement is already being taken by the ADR-0029 slice (the repair-event half of the
evidence log). This ADR names what that log *is for*: it is not only the raw data for the emergence
study (which panes earn default status) but also the raw data for **language evolution** (which syntax
capable agents author correctly). One instrument, two readings.

## Consequences

- **The language gets a feedback loop it did not have.** Syntax decisions become falsifiable against a
  corpus of real authoring attempts by capable agents, rather than resting on the language designer's
  intuition about what "reads well." ADR-0009 made the screen empirical; this makes the grammar
  empirical.
- **Prompt and grammar are two knobs on one measurement, and the telemetry says which to turn.** A
  cluster a prompt profile dissolves was a teaching gap; a cluster that survives good teaching is a
  grammar gap. The repair-success-rate-conditioned-on-error-message statistic is what separates them,
  so we stop guessing whether the fix belongs in the prompt or the parser.
- **Error messages become graded artifacts.** Because repair-success is measured *conditioned on the
  message*, the parser's repair-guiding errors (the a85b-workstream's messages the ADR-0029 loop
  surfaces) are themselves under measurement — a message capable models cannot recover from is a
  defect, visible in the aggregate rather than only in a frustrated transcript.
- **Fail-closed honesty is preserved, not traded away.** Measuring the grammar never means the parser
  guesses. The parser stays exact and fail-closed; the loop repairs by *re-asking the model*, not by
  loosening acceptance. Grammar evolution changes the frozen spec deliberately (a new language
  version), never the parser's willingness to accept malformed input at runtime.
- **The evidence log is strategy-bearing and stays private.** The malformed-input signatures and the
  requested-View sequences reveal how strong models author and perceive; like the ADR-0029 emergence
  log they are private application data (ADR-0029 Open q7), and only aggregate cluster statistics are
  publishable.

## Open questions

1. **The signature normalization.** What is the right fingerprint of a malformed author — enough to
   cluster "the same mistake" across models, without being either the raw strategy-bearing bytes or so
   lossy that distinct errors collapse together? The v1 slice records the parser error class + a coarse
   structural signature; the clustering function is a measurement question to refine as the corpus grows.
2. **The promotion threshold.** How dominant and how cross-model must a cluster be, and how thoroughly
   must a prompt fix have failed, before a grammar change is warranted? Recommended: owner-gated from
   the aggregate (like the ADR-0009/0029 freezes), never an automatic rule — but the threshold's shape
   is unsettled.
3. **Backward compatibility of a grammar relaxation.** A relaxation that accepts a previously-rejected
   form must not change the meaning of any form that already parsed (a superset extension only), and
   must mint a new language version so existing armed-document hashes are unaffected. The exact
   compatibility discipline for a grammar version bump is to be pinned when the first relaxation is
   proposed.
4. **Whether repair itself contaminates the capability signal.** A model that authors invalid then
   repairs to valid under a good error message has demonstrably *more* capability than one that cannot
   recover — but it also spent more attention (ADR-0029 §7 folds repair tokens into `ev_per_ktoken`).
   Whether repair-success should be reported as its own capability axis, distinct from first-attempt
   validity, is an evaluation-design question the emergence study will have to settle.