UNPKG

kestrel.markets

Version:

A typed, token-efficient language + runtime for agentic trading: agents author bounded plans, the runtime fires them at the tick. CLI + typed library + MCP server.

158 lines (139 loc) 17.6 kB
# The System Profile — the pinned run-identity schema for a benchmark run > **Status:** v1.1 — **owner-approved 2026-07-13 with tightening** (the m9i.2 owner review; no > redesign). The System Profile is the *identity* half of kestrel-m9i.3's acceptance; the > statistical protocol half (EVALUATION.md + platform ADR-0018 seasons/DSR) was owner-approved the > same day. The review's three BLOCKING items are **implemented** (CellKey keys on the ConfigId — > the *cell* ConfigId, i.e. full minus the `fillSeed` replicate axis, so behavior splits cells while > seeds pool as replicates, ADR-0016 §3; the `face` axis; adapter/harness mandatory-for-certified); > the remaining items are > REQUIRED before the first certified cohort and carry `[PLANNED]` markers with their beads > (kestrel-m9i.23/.24/.25/.26, kestrel-m9i.8). > > **Freeze precondition (owner, verbatim in spirit):** no System Profile freezes while the fill > stack (kestrel-9gu.9 / kestrel-9gu.10 / kestrel-xwf) is open — `fill_model` is a profile field, > and freezing now would pin a known-invalid judge. ## Purpose The System Profile is the owner-approved set of fields that must be **declared and frozen** for a graded Blotter to be comparable, reproducible, and honest — the complete identity of *what was run*, stamped on every benchmark result. It reuses the identities that already exist in code (the a57.14 `ExperimentalEnvelope`, `ConfigId`, `FillModelStamp`, `SimRunId`/`CellKey`) rather than minting a parallel schema — **one definition, never a second one** (ADR-0013(d)). A profile that is incomplete on any *mandatory* field fails the Blotter closed to **provisional**, never silently defaulted (a57.14; the Certification non-negotiable). ## The field schema Provenance classes: **CFG** folds into `ConfigId` (content-hashed, `src/session/config.ts` `deriveConfigId`); **ENV** = a57.14 `ExperimentalEnvelope` field (also a CFG input — the ConfigId is a superset of the envelope); **ATT** = attested-only, off the deterministic record path (injected / verified, never hashed into `ConfigId`); **DER** = derived content-address of a record. `[PLANNED …]` marks a field an m9i AC asks for that has **no code seam yet** — named with its bead. ### Layer 1 — AGENT IDENTITY (who authored) | Field | Class | Seam / justification | |---|---|---| | model id | ENV·CFG | `envelope.model` / `config.model` (a57.14; ADR-0013(d)) | | version / snapshot | ENV·CFG | `envelope.version` / `config.version` | | training_cutoff | ENV·CFG | `envelope.training_cutoff` — basis for model-relative holdback (a57.14). **Load-bearing**: liveAgent runs without it stamped *provisional*. Registry cross-check against the platform model registry (fence `MODEL_KNOWLEDGE_CUTOFF`), fail-closed on unknown model ids, with the **pinned month-boundary rule** — a tape date is post-cutoff iff *strictly after the last day of the declared month*. **Implemented (kestrel-m9i.23)** — the fence + the sole month-boundary rule live in `src/blotter/model-registry.ts` (`MODEL_KNOWLEDGE_CUTOFF` / `isPostCutoff` / `checkTrainingCutoff`, mirroring the platform `fence.ts`); the a57.14 projector cross-checks a complete envelope and fails it closed to *provisional* with a typed reason on an unknown model id or a declared-vs-registry mismatch (never a silent default, never a rewrite). | | provider / API | ENV·CFG | `envelope.provider` / `config.provider` (a generic label, NEVER a credential — ADR-0013(e)) | | endpoint class || dual-harness rung: Bedrock inference-profile vs direct-provider vs gateway-forbidden (ADR-0013(c)). `[PLANNED kestrel-m9i.2 — no envelope slot today]` | | **face** | ENV·CFG | `envelope.face` / `config.face` — the interface the author reached the model through, a **closed vocabulary** `http \| sdk \| cli \| mcp` enforced fail-closed by the projector (unknown/absent ⇒ provisional, never coerced). The m9i.7 tournament axis. Declared by the CALLER (who knows the transport); a harness never fabricates it. **Implemented (m9i.2 owner tweak 2)** | | **adapter / harness + version** | ENV·CFG | `envelope.adapter` + `envelope.adapter_version` / `config.adapter` + `config.adapterVersion` — the harness the our-harness vs Claude Code vs Codex vs OpenCode comparison pivots on. **Mandatory-for-certified** BEFORE the harness tournament; `liveAgent` stamps its own identity (`kestrel-live-agent`/`v0`) when the caller pins none. **Implemented (m9i.2 owner tweak 3)** | | prompt-profile hash | ENV·CFG | `envelope.prompt_hash` / `AuthorPolicy.prompt_sha256` — a HASH, never prompt content (the provenance fence, ADR-0013(d), AGENTS.md §7) | | tool binding | ENV·CFG | `envelope.tool_policy` / `config.toolPolicy` (e.g. `no-tools`) | | tokenizer | ENV·CFG | `envelope.tokenizer` / `config.tokenizer` (RenderingIdentity) | | rendering_identity | ENV·CFG | `envelope.rendering_identity` (distinct from `format`). **Drift-proof binding** — a renderer revision folded into ConfigId, so a renderer change mints a new grid column (kestrel-commit is ATT-only and does not). **Implemented (kestrel-m9i.24)** — DERIVED from the ONE-place renderer epoch `RENDERER_REVISION` (`src/frame/renderer-revision.ts`, `kestrel-renderer/<N>`, decoupled from `PROTOCOL_VERSION`/the ATT commit — a renderer-behavioral change MUST bump it) via `deriveRenderingIdentity` (`src/session/config.ts`) as the content-addressed handle `sha256:${sha256("kestrel.rendering/" + rev)}`; a real run sets `renderingIdentity = deriveRenderingIdentity()`, so a revision bump ⇒ a new handle ⇒ a new ConfigId/CellKey (ADR-0013(d)). Additive: the current renderer reproduces the current stamped value byte-for-byte, so every existing ConfigId/CellKey/conformance-root/grid pin is unchanged. | | format | CFG | `config.format` — ConfigId axis only, not an envelope field (config.ts) | | temperature / thinking policy | CFG | `config.temperature` / `config.thinkingLevel` — ConfigId axes only. **model-default honesty**: where a provider rejects the setting (the newest Sonnet rejects `temperature`, ai-sdk-client) the profile records a `model-default` sentinel, never a fabricated `0` (m9i.1 "never quietly turn one config into another"). `[PLANNED kestrel-m9i.2 — sentinel]` | | cache policy | CFG | `config.cachePolicy` — the live harness's per-session conversation/KV-reuse + provider prompt-cache regime (`stateless-redraw \| conversation \| conversation-cached`, ADR-0013 / kestrel-rul). A **ConfigId axis only**, NOT an envelope field: caching is transport-side + off the graded path, so two policies grade byte-identically yet mint distinct grid columns (the m9i.5 A/B). `liveAgent` stamps `conversation-cached` when the caller pins none; TurnCapture records provider-native `cacheReadTokens`/`cacheWriteTokens` per turn as the m9i.5 evidence. **Implemented (kestrel-rul)**; economics measured by `[kestrel-m9i.5]` | | deadline / timeout policy | — | generation/deadline policy (m9i.2 AC). `[PLANNED kestrel-m9i.2 / kestrel-m9i.6]` | | wake_source | ENV·CFG | `envelope.wake_source` / `config.wakeSource` (e.g. `structural-cadence`) | | token accounting source | ENV·ATT | `envelope.token_accounting` / `config.tokenAccounting` — INJECTED, off the record path (a57.14). **Load-bearing**: absent ⇒ provisional last night | ### Layer 2 — RUNTIME IDENTITY (under what judge) | Field | Class | Seam / justification | |---|---|---| | kestrel version / commit | ATT | reproducibility pin (m9i.2). `[PLANNED kestrel-m9i.2]` | | fill model + calibration id | DER | `FillModelStamp {name, version, calibration_sha}` (src/bus/types.ts; ADR-0006) — public tag = `name`/`version`, private constants by hash = `calibration_sha` (ADR-0016). CellKey `fill_model` axis | | fair / price model tag | DER | travels inside the fill judge today (`maker-fair-v1`, RUNTIME §6). A distinct price-model tag `[PLANNED kestrel-m9i.2]` | | judge version | DER | fill judge = `FillModelStamp.version`; the a57.2 Grade-protocol version `[PLANNED kestrel-a57.2]` | | graded-bus sha256 | DER | `SimRunId = sha256(serializeBus(gradedBus))` = `blotter.session.bus.sha256` (run-identity.ts) | | input-tape sha256 | — | no tape-content hash on the Blotter today; tape axis is `session.instruments` (run-identity.ts DESIGN NOTE). `bus_sha256` already exists in the sessions manifest — a small wire. **REQUIRED before the first certified cohort** (owner item 4). `[PLANNED kestrel-m9i.8]` | | corpus tier (practice / holdback) | ENV·CFG | `envelope.corpus_tier` / `config.corpusTier`; practice-vs-season wall (platform ADR-0018). Typed to the closed ADR-0018 addendum three-tier vocabulary `CORPUS_TIERS` (`public / semi-private / private`, src/bus/types.ts), enforced fail-closed at projection exactly like `face` — a declared-but-unknown tier (a typo, or the retired `public-baseline`) refuses to PROVISIONAL, never silently certifies (kestrel-m9i.26). The `corpusTierWall` classifier maps `public → practice` (freely trainable, non-ranking) and `semi-private`/`private → holdback` (sealed forward window, event-date > training_cutoff, ranking-eligible) | | fillSeed(s) | CFG | `runSeed`; seed ensembles (`seed+i`) are **replicates**, not extra samples (ADR-0016 §3). **fillSeed is a REPLICATE axis: it folds into `ConfigId`/`SimRunId` (each seed is a distinct RUN) but is STRIPPED from `CellKey`** (`cellConfigId`, config.ts) — so seed-only variants are N replicates in ONE cell (MC error via CellAggregate), never distinct columns. Lands in `AgentConfig` `[kestrel-9gu.8]` | | ConfigId | DER, **stamped** | `deriveConfigId(canonical config)` (config.ts) — superset of the envelope; the FULL, seeded RUN identity. Since the m9i.2 owner tweaks the driver stamps it on the graded META (`config_id`) and the projector carries it to `session.config_id` | | SimRunId / CellKey | DER | per-run + `(tape × fill_model × config)` cell (run-identity.ts). **CellKey keys the config axis on the CELL ConfigId** = the full ConfigId with the `fillSeed` replicate axis STRIPPED (`cellConfigId`; stamped as `meta.cell_config_id` → `session.cell_config_id` only when it diverges from `config_id`, i.e. a seed was present; else `cellOf` reads `config_id`) — temperature-only variants are distinct cells, seed-only variants are replicates in one cell (m9i.2 owner tweak 1 reconciled with ADR-0016). `SequenceCellKey` (5zl.16.5) composes per-session keys, absorbing the same axis | | InstanceRunId | — | `sha256` of ordered per-session SimRunIds for a sequence (ADR-0018). `[PLANNED kestrel-5zl.16]` | ### Layer 3 — DIVISION / GOVERNANCE (which contest, sealed how) | Field | Class | Seam / justification | |---|---|---| | division (CONTROLLED vs OPTIMIZED) | ATT | the two m9i estimands, never averaged (EVALUATION.md; ADR-0013(d) — selector = prompt-profile variant-set). A stamped division field `[PLANNED kestrel-m9i.2 / m9i.8]` | | season id | ATT | governed season registry (platform ADR-0018). `[PLANNED kestrel-m9i.8 / m9i.9]` | | submission freeze hash | DER·ATT | content-addressed frozen submission before window open (platform ADR-0018; m9i.8 AC "content-addressed, human-approved"). `[PLANNED kestrel-m9i.8]` | | T-5 / cutoff attestation | ATT | T-5m date-blind briefing + few-looks cutoff (EVALUATION.md; benchmark tiering). `[PLANNED kestrel-m9i.11]` | | training_cutoff ↔ tape-date firewall | ATT | honest-tier only after cutoff, date-blind — the contamination fence (EVALUATION.md; `training_cutoff` is the gating basis, a57.14). Enforcement `[PLANNED kestrel-m9i.8 / m9i.9]`; the month-boundary rule is pinned in kestrel-m9i.23 | | **rankable** (honest-tier flag) | DER | a DERIVED flag = certified AND post-cutoff AND date-blind AND (season) frozen — ends "certified" doing double duty as leaderboard eligibility (the mislabeled pre-cutoff rows). A pure read of stamped fields, never a second stamped verdict. Landed in kestrel-m9i.25: `blotter.rankability` (four legs, per-leg NAMEABLE reasons, fail-closed UNKNOWN ⇒ not rankable) is derived at projection in `src/blotter/project.ts`; the leaderboard surfaces it as a `RankableBreakdown` DISTINCT from the certification breakdown (`src/session/grid.ts`). | ## Mandatory for **certified** vs **provisional** "Certified" is the existing Blotter Certification verdict (CONTEXT.md), which a57.14 already flips to **provisional** on any incomplete declared identity — this profile does not mint a second eligibility flag. (Leaderboard *eligibility* becomes the separate DERIVED `rankable` flag — kestrel-m9i.25 — so "certified" stops doing double duty.) - **Mandatory for a certified benchmark row** — the complete a57.14 envelope (**all 14 fields**: provider, model, version, training_cutoff, prompt_hash, tokenizer, tool_policy, rendering_identity, wake_source, token_accounting, corpus_tier, **face**, **adapter**, **adapter_version** — the last three per the m9i.2 owner tweaks, mandatory BEFORE the m9i.7 face/harness tournament; `face` must additionally be inside the closed `http|sdk|cli|mcp` vocabulary) **+** the `FillModelStamp` **+** the graded-bus `SimRunId`. Any missing/empty one ⇒ **provisional**, with a reason naming the field. - **Mandatory additionally for a *season* row** — division stamp, season id, submission freeze hash, and the T-5 / holdback firewall attestation (all `[PLANNED]`; provisional until they land). - **Not a benchmark row at all** — a NO-LLM Backtest declares no envelope (`declaresEnvelope` false) and stays `certified` as a *record*, but authors nothing and never ranks (config.ts; a57.14). ## How each field folds into ConfigId (or is attested-only) - **Into `ConfigId`** (content-hashed, any edit mints a new grid column — ADR-0013(d)): the whole canonical `AgentConfig` — model, version, training_cutoff, provider, prompt_hash, tokenizer, rendering_identity, tool_policy, wake_source, corpus_tier, face, adapter, adapterVersion (the 13 CFG-ENV axes) **plus** format, temperature, thinkingLevel, label (CFG-only, non-envelope). The `ConfigId` is a **superset** of the `envelope` identity, and since the m9i.2 owner tweaks it ALSO lands on the Blotter (`session.config_id`, stamped by the driver as `meta.config_id`). - **CellKey axes** (run-identity.ts): tape (`instruments`) × fill_model (`FillModelStamp`) × config (the **cell ConfigId** = the full ConfigId with the `fillSeed` replicate axis STRIPPED — via `session.cell_config_id` when stamped, else `session.config_id`). m9i.2 owner tweak 1: temperature-only variants are DISTINCT cells; reconciled with ADR-0016 §3: **seed-only variants are REPLICATES in ONE cell**, never distinct columns (a seed is measurement noise, not behavior). A pre-tweak Blotter with neither id falls back to the envelope-identity axis as a legacy cell, never pooled with a ConfigId-keyed cell. A future BEHAVIORAL config axis is absorbed automatically once it enters `AgentConfig`; a future REPLICATE axis is added to `cellConfigId`'s strip-list. - **Attested-only, never hashed into ConfigId** (off the deterministic record path): token_accounting, wire usage / latency / wall-clock, cost-table version, cache-hit counts, kestrel commit, division, season id, freeze hash, T-5 attestation. Verified and stamped, but a re-projection must byte-match regardless of them (ADR-0011). ## Example — last night's sonnet-4.6 Bedrock run (illustrative values) ``` AGENT IDENTITY model us.anthropic.claude-sonnet-4-6--v1:0 version us.anthropic.claude-sonnet-4-6--v1:0 training_cutoff 2025-11 # gating basis; run tapes must post-date it provider amazon-bedrock # endpoint class: bedrock-inference-profile [PLANNED] prompt_hash sha256:(baseline-v1 system bytes — a hash, never the prompt) tokenizer claude rendering_identity ascii-v1 format ascii temperature model-default # Sonnet rejected an explicit value — honest, not 0 [PLANNED] thinkingLevel none tool_policy no-tools wake_source staleness-backstop face sdk # closed vocabulary http|sdk|cli|mcp (m9i.2) adapter kestrel-live-agent · v0 # stamped by the harness itself (m9i.2) token_accounting { input, output, thinking } # injected from provider usage (ATT) RUNTIME IDENTITY fill_model strict-cross-v1 · calibration_sha sha256:(none-sentinel: floor judge) corpus_tier public # practice tier — non-ranking; closed vocab public|semi-private|private (m9i.26) graded-bus SimRunId <blotter.session.bus.sha256> ConfigId <sha256(canonical AgentConfig)> # stamped: meta.config_id → session.config_id (full, seeded — RUN identity) CellKey { tape:, fill_model:, config:<cell ConfigId = full minus fillSeed> } # m9i.2 tweak 1 ⊕ ADR-0016 §3 kestrel commit <HEAD> # [PLANNED kestrel-m9i.2] DIVISION / GOVERNANCE division CONTROLLED (baseline-only) # [PLANNED stamp — kestrel-m9i.2/m9i.8] season id — (practice run, not a season entry) # [PLANNED kestrel-m9i.8] Verdict: this profile is COMPLETE on the 14 envelope fields + fill_model + SimRunId ⇒ the Blotter certifies. It is a practice-tier CONTROLLED result; certified but NOT `rankable` (pre-cutoff tape — kestrel-m9i.25), and it can NEVER appear on a season leaderboard (platform ADR-0018) until frozen into a sealed-forward season submission. ```