akm-cli/docs/architecture/persisted-data-compat.md

akm (Agent Knowledge Manager) — a portable, local-first capability library for AI agents. Discover, load, share, and improve reusable skills, scripts, workflows, and knowledge across any shell-capable coding agent, including Claude Code, OpenCode, and Cur

# Persisted-Data Compatibility Contract

akm persists state in twelve-odd formats across releases: config, three SQLite
databases, task and workflow documents, native scheduler rows, and a few smaller JSON/text payloads. Each has grown its own
versioning and its own failure behavior. This document is the one place that
states what a reader owes data an earlier release wrote, and inventories how
close each format is to that contract today.

## The contract

A reader must tolerate anything an earlier release wrote: convert the old
shape in memory, warn once, and name `akm migrate apply` as the on-disk
cleanup path — the migrator is where an explicit, backed-up rewrite happens,
never a precondition for reading. The only refusal a reader is allowed is data
written by a release *newer* than itself, and that refusal must name the
remedy (upgrade akm). One format is a named exception: task sources
(`tasks/*.yml`) are read only in the current grammar, and an older file is
refused on its own, naming `akm migrate apply` (#987; see the Task source
row). Bundle content is never rewritten just to make it
readable again by an older or current binary. Every format-version bump must
ship with its old shape covered by the upgrade rehearsal gate
(`tests/integration/upgrade-rehearsal/`). No config object is strict: an unknown key at any depth is kept in memory
and named once, round-trips through ordinary writes, and is dropped by
`akm migrate apply`.

## Formats

| Format | Where written | Version marker | Older data | Newer data | Gate |
| --- | --- | --- | --- | --- | --- |
| `config.json` | `src/core/config/config.ts` | `configVersion` field | `configVersion` is read, never gated on (`readConfigVersion`, `src/core/config/config.ts`): a missing field or `"0.9.0"` — the only value akm has shipped — loads silently, and any other value is named once and read as `0.9.0`; like the `extraParams` keys, the legacy `stashDir`/`sources[]`/`installed` shape is converted rather than dropped — `migrateLegacySourceShape` (`src/core/config/legacy-source-shape-shim.ts`) folds it into `bundles`/`defaultBundle` in memory on every load; `akm migrate apply`'s `configFile` step (`normalizeConfigFile`) persists all of it — `configVersion: "0.9.0"`, the converted source layout, the lifted `extraParams`, unknown keys dropped — to the file once, under a backup | Same as older: the value is named once and the file is read as `0.9.0`; there is no `UNSUPPORTED_CONFIG_VERSION` refusal. `akm health`'s `binary-config-skew` advisory (`src/commands/health/config-skew.ts`) is what says "a newer akm wrote this config, upgrade this install" | Upgrade rehearsal gate (`tests/integration/upgrade-rehearsal/`); `tests/config-v09.test.ts` |
| `state.db` | `src/core/state/migrations.ts` (append-only `MIGRATIONS` registry, applied by `src/storage/sqlite-migrations.ts`) | `schema_migrations` ledger table | `openStateDatabase` opens one connection (mkdir, open, standard pragmas), reads the ledger on it, and runs every migration not yet applied forward in one transaction; a missing/empty/table-less file is simply a fresh database. Before a migration that drops schema (`018-drop-dead-lane-schema`, `028-improve-ledger`; `STATE_MIGRATION_SAFETY_BY_ID`) runs against a database that already has migrations applied, the file is copied beside itself to `state.db.pre-<migration id>.bak` via `VACUUM INTO` on the same connection | An older binary opening a `state.db` migrated by a newer akm continues with the schema it knows and warns once (`warnNewerStateLedger`, `src/core/state-db.ts`); it does not refuse; only a ledger that diverges from this akm's registry is refused (`assertMigrationLedger`) | Upgrade rehearsal gate (`tests/integration/upgrade-rehearsal/`) |
| `index.db` | `src/storage/repositories/index-schema.ts` (`ensureSchema`; `index-entry-schema.ts` for the DDL and the layout constant) | `index_meta.version` = `CANONICAL_INDEX_DB_VERSION` (26), a layout marker | Readers (`openExistingDatabase` / `openReadonlyExistingDatabase`) serve it as-is and name `akm index` once (`checkIndexLayout`); the writable opener migrates in place — added columns via `ALTER TABLE`, a one-time FTS rebuild from `entries` for the layout-23 content-copy tables, and for layout 24 the drops of the sqlite-vec mirror `entries_vec` (when sqlite-vec loads; dropping a vec0 table needs it) and `entry_fragments_fts`, with `entries.search_text` hashed into `embed_hash` before the column is dropped so no vector is re-embedded, and for layout 25 the declared links (`asset_links`, #935) derived from each entry's stored `document_json` with no file read (only the directories holding workflows and tasks, whose targets no earlier layout stored, lose their incremental cursor and re-drain on the next `akm index`) — never dropping `embeddings`, `utility_scores*`, or `llm_enrichment_cache`. It drops the lazy-extraction queue `graph_extraction_queue`, retired in 0.9.17, and, on any layout, the LLM entity graph (`graph_meta`, `graph_files`, `graph_file_*`) and its `llm_enrichment_cache` rows, retired in 0.9.17-alpha.9 — an older release's `CREATE TABLE IF NOT EXISTS` recreates the graph tables (empty) if it ever opens this index again. It also deletes `llm_enrichment_cache` rows at the default `cache_variant` — the metadata-enhance pass, retired in 0.9.17-alpha.9 — on any layout; memory inference uses its own named variant and is unaffected. A migration leaves `index_meta.vacuumPending`, and the next `akm index` VACUUMs the freed pages. An `entries` table older than layout 21 has its entries-keyed tables recreated: one that lacks a column this release reads, or that still has a transitional column layout 21 removed (`entry_key`, `dir_path`, `stash_dir`, `entry_json`, `entry_type`), as the table 0.9.1 wrote does, with `document_json` empty on every row. A reader finds such a table unservable (`hasCurrentEntriesTable`), so a read rebuilds it inline. Only `SQLITE_CORRUPT` deletes and rebuilds the file (#865) | Refused by readers and the writable opener alike (`INDEX_SCHEMA_INCOMPATIBLE`, "Upgrade akm to use this index.", `newerIndexLayoutError`), leaving the file untouched: a newer layout may lack a column this akm reads, as layout 25 lacks `search_text` | `tests/integration/indexer/index-layout-migration.test.ts` (layout-23 and layout-24 fixtures); `tests/integration/previous-release-corpus.test.ts` (the layout-25 index 0.9.17-alpha.7 wrote, `index-v25.sql`, and the layout-20 index 0.9.1 wrote, `index-v20.sql`); `tests/integration/storage/index-generation-read-boundary.test.ts`; upgrade rehearsal gate (`tests/integration/upgrade-rehearsal/`) |
| Task source (`tasks/*.yml`) | `src/tasks/source/parse-task-source.ts` | `version: 4` (`TASK_SOURCE_V4_VERSION`) | **The named exception to the contract (#987): the runtime reads only v4.** `version: 2` or `version: 3` throws `TASK_SCHEMA_VERSION_UNSUPPORTED`, and a declared `version: 4` document whose `schedule[]` still carries 0.9.15's retired `enabled` key throws `TASK_SOURCE_INVALID`; both messages name `akm migrate apply`, the one place the conversion happens (`scripts/akm-migrate/migrate/task-files.ts`, planning every v2/v3/v4 file straight to v4 with the pure planner `src/tasks/source/task-to-v4.ts`, rewriting under a backup; `akm upgrade` runs it after an install). It covers every stash the runtime reads tasks from: each enabled configured bundle, and the stash `AKM_BUNDLE_DIR` selects when no bundle names it; the `akm-task` probe still recognizes a root of v2/v3 task files, so one without a recorded adapter is found. Every caller reports the refusal for that one file: `akm task sync` keeps reconciling every other task and leaves the file's installed row as it is | Any other `version` throws `TASK_SCHEMA_VERSION_UNSUPPORTED` naming `akm migrate apply` | Upgrade rehearsal gate (`tests/integration/upgrade-rehearsal/`); `tests/integration/previous-release-corpus.test.ts` (each older shape is refused at runtime and converted by the migrator) |
| Workflow IR (frozen plans) | `src/workflows/freeze/freeze.ts`, read back in `src/workflows/runtime/run-plan.ts` (`readRunPlan`, `decodeWorkflowPlan`) | `irVersion` (`WORKFLOW_PLAN_VERSION`, 6) on the plan and in `plan_ir_version`; `plan_hash` alongside | Every workflow compiles to one plan type. `decodeWorkflowPlan` reads the irVersion 4 and 5 shapes (their `sourceReadSet` and host-identity fields included) as well as the current one, and keeps keys it does not know. Neither the version nor the hash gates a read: a stored plan that decodes runs whatever `irVersion` froze it, with one warning; one that cannot be decoded is abandoned by `akm workflow run` (status `failed`) with a message naming `akm workflow run <ref>` to start afresh — a status change, never an exception; `status`/`list`/`abandon` never read the plan | A newer akm's plan that still decodes runs the same way; one that does not decode is left untouched and refused with a message naming "Upgrade akm" (`readRunPlan`'s `newer`) | `tests/workflows/run-plan.test.ts`; `tests/integration/workflows/frozen-plan.test.ts` — the rehearsal builds no workflow run |
| Native scheduler rows (crontab, launchd plists, Task Scheduler) | `src/tasks/backends/cron.ts`, `launchd.ts`, `schtasks.ts`; the invocation grammar in `src/tasks/scheduler-invocation.ts` | The row's own invocation shape: current rows set their environment inline (a cron `VAR=value` prefix, a plist `EnvironmentVariables` entry, a PowerShell `$env:` assignment) — always `AKM_BUNDLE_DIR`, the syncing shell's working stash — and carry no `--scheduler-context`; 0.9.0 – 0.9.17-alpha.6 rows carry `--scheduler-context <descriptor>` holding the same values; older rows carry neither. A cron command over 1,000 bytes runs an akm wrapper script (`sh <log dir>/.akm-cron-wrapper-…sh`) that holds the same line | Every shape is listed and parsed (`parseScheduledInvocationArgv`; a spilled cron row from its wrapper script): a row inside akm's own `# akm:task` markers from before `--scheduler-context` (#881), and a `--scheduler-context` row, which keeps firing (the CLI applies its descriptor, below) until the next plain `akm task sync` rewrites it in place — an update that keeps the row's launcher and schedule, never an add or remove. The primary bundle proves a row by the `AKM_BUNDLE_DIR` it carries, inline or in its descriptor (#846); any other bundle owns rows by name. Rows that still carry the pre-0.9.17 workflow execution-evidence marker (`# akm:workflow-evidence` in the crontab, an `akm-workflow-evidence` plist comment or PowerShell literal) are listed like any other and rewritten once without it. Plain `akm task sync` keeps an installed row's own launcher (`installOptionsFor`, `src/tasks/scheduler-sync.ts`) and recomputes its environment; `akm task sync --rebind` repoints the launcher | A row inside akm's markers, `com.akm.task.` label namespace, or `\akm\` task folder whose argv this binary cannot parse (for example one a newer akm wrote) is not listed: sync leaves it alone unless an enabled source renders a row with the same native id, which then replaces it — the row is derived state and its source is the record. A crontab whose akm markers are malformed is refused unmodified (`parseBlocks`, `cron.ts`) | Upgrade rehearsal gate (`tests/integration/upgrade-rehearsal/`: the previous release's `--scheduler-context` rows fire under the candidate, then `task sync --dry-run` shows them only as updates); `tests/tasks-sync.test.ts` (`--scheduler-context` rows and spilled rows are updates, never adds) |
| Scheduler-context descriptor (`$DATA/tasks/context/<sha256>.json`, written by 0.9.0 – 0.9.17-alpha.6) | No longer written: a row carries its own environment (#987). Read by `src/tasks/scheduler-invocation.ts` (`readLegacySchedulerContext`) | `version: 1`; the environment key set | When a row that names one fires, the CLI reads the file as plain JSON and applies its `AKM_*_DIR` keys and, in one written before 0.9.17, `PATH`; nothing else about the file (name, mode, owner, link) is checked. Sync reads its `AKM_BUNDLE_DIR` to attribute the row. A plain `akm task sync` that reconciles the row rewrites it without the argument; a row a sync leaves as it is (its source failed, a `--bundle` sync did not cover it) keeps naming the file | A key this binary does not know is ignored. A file that is gone or is not JSON fails the run with `INVALID_CONFIG_FILE` naming `akm task sync`; sync cannot attribute such a row to the primary bundle and leaves it in place, and `akm task prune` reports it as `invalid-context` | `tests/tasks-scheduler-invocation.test.ts`; `tests/integration/scheduler-context-launcher.test.ts`; upgrade rehearsal gate (`tests/integration/upgrade-rehearsal/`) fires the previous release's rows under the candidate before and after its plain `task sync` |
| Proposal `metadata_json` | `src/storage/repositories/proposals-repository.ts` | No explicit version field; presence/absence of the `changes` key | Rows from before the `changes` key existed (~89% of archived rows on real installs) are treated as a known legacy gap — decoded with an empty change list rather than thrown (`proposals-repository.ts:47-56`). 0.9.17-alpha.9 adds a consolidate pair-pass `retire` proposal's `changes[0].op: "delete"` (no `after`, already a valid `FileChangeOp`) and its `retirement` / `retiredArchive` / `promotionSource` / `promotionSourceHash` metadata keys, plus the retirement continuity check's `retirement.continuityRisk` (rule R3, item 1) — a person can accept a proposal carrying it by id even though it is excluded from bulk accept; an older reader's `validatePresentMetadata` does not enumerate these keys, so it neither inspects nor rejects them — the row decodes with them silently absent, same as any other field an older release predates. 0.9.19 adds `reviewHistory` (the rejections `akm proposal reopen` undid, oldest first, each with its gate verdict) the same way: a proposal reopened by 0.9.19 and read by an older release is an ordinary pending proposal with its history absent, and that release drops the key if it rewrites the row. The quality judge's evidence, `scores` and `judgeReason`, is added inside the existing `gateDecision` object (Unreleased): an older reader's `validatePresentMetadata` checks only the gate fields it knows and carries the object through whole, so the evidence survives a rewrite by an older release unread. An older release also counts a pending proposal's age (retention expiry, `--older-than`) from `createdAt`, so after a downgrade a reopened proposal can expire at the next `akm improve` run (a retire proposal never expires), or be swept by a scheduled `--older-than` bulk action or drain | 0.9.17-alpha.8 and earlier predate the retire proposal shape entirely — they were never taught to read a delete-primary change with no `after`. `akm proposal show` / `diff` / `drain` exit 70 on one, and drain's own nightly pre-pass failing on the first pending retire proposal it meets stops that run's auto-promotion for the whole stash, not just that one proposal. **Downgrade gate:** every pending retire proposal must be accepted or rejected (`akm proposal accept` / `reject`) before downgrading to 0.9.17-alpha.8 or earlier — once decided it is archived, out of the pending set `show`/`diff`/`drain` iterate, and no longer trips this. **Gap** otherwise (no code-level guard; this is operator guidance only). Downgrading also revives the starvation the second review round's B1 fixed forward-only: 0.9.17-alpha.8's `retrieval-scope.ts` counts a `consolidate-pair` ledger row as processed, so the pair pass's own nightly attempts crowd new material back out of every other improve lane's scope, the same way they did before B1 — measured on the real bundle, two alpha.9 nights left only 341 of 2,183 new-only assets still in scope once read under alpha.8. **Gap** here too (the ledger `source` string carries no version of its own; operator guidance only — do not downgrade across a pair-pass run without expecting this). | `tests/proposal-repository-pure.test.ts` — unit only; the rehearsal builds no proposal |
| `improve_ledger` rows | `src/storage/repositories/improve-ledger-repository.ts` | No version field of its own; the `content_hash` column (migration `029-improve-ledger-content-hash`) | A consolidate promotion row with no `content_hash` is held by its clock alone (`next_eligible_at`). 0.9.19 records `content_hash`, and no `next_eligible_at`, on a consolidate promotion once it is accepted or rejected, and holds the memory until its body hash differs (`isContentDrivenRow`); a promotion decided by an older release has no hash and keeps its old window (an accepted one none, a rejected one 7 days) | An older release reads a row by `next_eligible_at` alone and ignores the hash, so after a downgrade a memory whose promotion 0.9.19 decided is offered to the promote pass again at once, a rejected one sooner than the 7 days the older release would have held it; the hash stays on the row until the older release records its next attempt on that memory, which clears it. **Gap** (no code-level guard; operator guidance only) | `tests/integration/storage/improve-ledger-repository.test.ts` |
| `task_history` metadata | `src/storage/repositories/task-history-repository.ts` | `metadataVersion` (currently `2`) | An absent `metadataVersion` is treated as a legacy row (58% of rows written by prior releases) and decoded best-effort; unknown fields are dropped harmlessly rather than round-tripped (`task-history-repository.ts:61-69`) | A `metadataVersion` newer than this binary's `2` is decoded best-effort as version 2 with a warning, rather than rejected (`task-history-repository.ts:85-89`) | Upgrade rehearsal gate (`tests/integration/upgrade-rehearsal/`) |
| Lock payloads | `src/core/file-lock.ts` | No version field; payload is a bare numeric pid string, or a JSON object carrying `pid` and optionally `launcherPid` (`extractLockIdentity`, `file-lock.ts`) | Any payload shape parses permissively: extra fields are ignored, a missing or unparseable `launcherPid` is simply absent, and a payload with no parseable `pid` probes as `stale` (`invalid_pid`, `probeLock`) and is reclaimed rather than rejected | Same permissive parse; a payload from a newer akm that adds fields is read the same way | `tests/integration/file-lock.test.ts` (describe "probeLock: launcherPid (#956)") — not the rehearsal, which does not contend locks |
| `.akm` residue | Documented in `docs/architecture/internals/storage-locations.md` | No version marker | Stale `.akm` directories or files from a previous layout are inert; nothing in current code reads or interprets them | N/A | none — gap (the closest coverage, `tests/migrate/dead-residue.test.ts`, tests the detection and removal that `akm migrate status` / `akm migrate apply` run, not that ordinary commands tolerate leftover `.akm` residue) |

## Adding a format or bumping one

1. Give the format an explicit version marker (a field, a table, a filename
   convention) if it does not already have one — "no marker" is not a
   compatibility strategy.
2. Write the reader so it converts an older marker in memory and warns once,
   per the contract above. Reserve throwing for data from a version this
   binary has never heard of, and name `akm migrate apply` (or the specific
   rebuild command, e.g. `akm index`) in the error.
3. Add the old shape as a fixture to
   `tests/integration/upgrade-rehearsal/` so a real previous-release
   binary's output is exercised, not just a hand-written fixture.
4. If the format is `config.json`, nothing more: unknown keys at any depth
   are kept and named once by the reader (`unknownConfigKeyPaths`), and
   `akm migrate apply` drops them.
5. Add or extend the row in the table above, including a `Gap:` note if the
   code does not yet meet the contract — do not leave a mismatch undocumented.