akm-cli
Version:
akm (Agent Knowledge Manager) — a portable, local-first capability library for AI agents. Discover, load, share, and improve reusable skills, scripts, workflows, and knowledge across any shell-capable coding agent, including Claude Code, OpenCode, and Cur
378 lines (344 loc) • 23.7 kB
Markdown
# Design: bundle organization & back-linking conventions
Status: accepted (conventions shipped in the bundle skeleton)
Author: akm
Date: 2026-07-11
> **Amended by the 0.9.0 surface decisions.** Two mechanisms this document
> records as closed are superseded; the sections below that narrate the
> 2026-07-12 amendments are kept as a historical record, not current guidance:
> - **A tooled rename** (`akm mv`, SPEC-7) — the command still ships, but only
> as an **Experimental** surface outside the stability contract: its
> inbound-ref rewriting is inverted relative to the body-ref grammar. A rename
> is delete plus create: the destination is a new identity and learned state
> does not follow it. The recommended forced-rename procedure is the manual
> one (move, fix inbound refs, `akm index`, `akm lint`). See
> [0.9.0-decisions.md D3](0.9.0-decisions.md#d3--renames-are-delete--create-akm-mv-ships-experimental).
> - **The ref-prefix filter** (SPEC-4) — the `<type>:<prefix>/` spelling is
> retired in favour of conceptId prefixes (`memories/projecta/`,
> `bundle//skills/`). See
> [0.9.0-decisions.md D4](0.9.0-decisions.md#d4--browse-uses-conceptid-prefixes-not-type).
>
> The no-rename default this document argues for is therefore *stronger* under
> 0.9.0, not weaker: no tool you may rely on makes a rename cheap.
## Problem
The bundle skeleton already ships per-type authoring conventions
(`facts/conventions/assets/<type>.md`) that tell an agent **how to write** each
asset type. Nothing told an agent **where to place** an asset (which
subdirectory) or **how to cross-link** it. Yet placement and linking are exactly
what determine whether the same knowledge is *retrievable* two sessions later, or
re-derived from scratch — and whether cross-project reuse happens or duplicates
accumulate.
The request that drove this: define conventions that instruct agents how to
organize assets under their type directories (subdirectories for projects,
domains, etc.) and how to back-link them to improve retrieval during agentic
tasks. This document records the analysis, a structured debate, and the resulting
conventions.
## What AKM's mechanics dictate (the ground truth)
Any organization scheme has to fit how AKM actually stores and retrieves. The
load-bearing facts, each verified in code:
1. **Path becomes the ref.** A file's subpath under its type dir is part of its
ref name — `memories/projectA/auth-tip.md` → `memory:projectA/auth-tip`. Works
for every type. A ref-prefix query syntax exists since SPEC-4 landed
(amended 2026-07-11): `akm search "memory:projectA/"` enumerates exactly
that subtree (typed, recursive, `/`-boundary exact) and a bare `memory:`
enumerates the whole type. At review time the idiom was false —
`sanitizeFtsQuery` strips `:` and `/` and `entry_type` is not an FTS
column, so the query degenerated to token noise (empirically tested; see
the Review round). `akm search "projectA" --type memory` remains the
token-match alternative. The path is
therefore the one facet you cannot express twice and cannot change without
breaking the ref (and every inbound reference to it).
2. **Retrieval is search, not browse — and folders are highly visible to the
ranker.** There is no folder walk at query time — agents `akm search`/
`akm curate`, then `akm show <ref>`. A subdirectory's real payoff: its tokens
join the FTS `name` column at the highest bm25 weight (10.0), always merge
into `tags` (since SPEC-2 landed; see fact 6), and match the cwd
project-context boost (`projectContextRankingContributor`, +0.2/token,
cap 0.5). No "indexing-confidence bump" for subdirectories exists in code.
3. **The FTS surface is `name`, `description`, `tags`, `hints`, `content`**
(`entries_fts`, `src/storage/repositories/index-schema.ts`). There is **no `project` field** —
a bare `project:` frontmatter value is invisible to search. Off-axis facets
must be `tags` to be retrievable.
4. **`xrefs:` fold into the FTS `hints` field for all types**
(`src/indexer/search/search-fields.ts:58`). Back-links are a *retrieval*
signal, not decoration — which means both too few and too many degrade
ranking.
5. **Project relevance is recovered at query time.** Ranking blends
a per-project usage signal — `scopedUtility * 0.7 + globalUtility * 0.3`
(`SCOPED_UTILITY_BLEND_SCOPED`, `src/indexer/search/ranking-contributors.ts`),
keyed by a cwd-anchor (a hash of the querying project root), not the asset's
path. It is NOT rename-proof for the asset: `entry_key` includes the full
name, so renaming a file mints a new entry row and orphans its accumulated
global + scoped utility history. As of 0.9.0 that is the defined semantics —
a rename is delete plus create, and no command preserves learned state
across one (see 0.9.0-decisions.md D3). A
reusable asset does not need the *project* baked into its path to rank well
inside a project — and a manual rename costs learned ranking.
6. **Directory (scope/domain) tokens always merge into `tags`** — since SPEC-2
landed (`extractDirTagsFromName`, `src/indexer/passes/metadata.ts`) they are
derived from the canonical ref subpath, so explicit `tags` no longer
suppress the scope token, both indexing walks agree, and the exact-tag
ranking boost fires for the path token without restating it. Filename
tokens are still auto-derived only when `tags` is empty (they already live
in the FTS `name` column and in aliases). The old explicit-tags footgun is
gone; existing installs pick the merged tags up on the next reindex.
7. **The entity/relation graph boost extracts from body content, not the path**
(`graph-extraction.ts`). Canonical entity naming must live in the prose.
8. **`category: convention|meta` facts are surfaced to every non-wiki authoring
flow** (`resolveStashStandards`); `facts/conventions/assets/<type>.md` is
surfaced type-scoped (`resolveTypeConventions`). This is the only channel that
reaches a browse-blind agent mid-task — so conventions must ship as facts.
9. **Non-wiki xref breakage is caught by `akm lint`, not at write time.** The
deterministic `missing-ref` check covers body refs, the `refs:` frontmatter
array, and (since SPEC-1 landed) the `xrefs:`/`supersededBy:`/
`contradictedBy:` frontmatter channels. A rename dangles inbound links
silently — nothing catches them until the next `akm lint` run flags them, so
run `akm lint` as the last step of any rename. (`akm mv` automates this, but
its rewriting is inverted relative to the body-ref grammar, so it ships
Experimental in 0.9.0 and does not substitute for the lint run — see
0.9.0-decisions.md D3.) Wikis are excluded from
`akm lint`'s directory sweep; the equivalent
orphan/broken-xref/broken-source/stale-index/uncited-raw structural checks
are implemented in the LLM Wiki adapter
(`core/adapter/adapters/llm-wiki-adapter.ts`, ported from the pre-0.9.0
`akm wiki lint`). **Amendment:** the `akm wiki` command family, including
`akm wiki lint`, was removed in the 0.9.0 bundle-adapter cutover (see
[wikis.md](../../guides/wikis.md)) — there is currently no CLI surface that
invokes these checks.
## Research inputs
Three parallel research briefs (Karpathy's LLM wiki; agent memory / RAG /
GraphRAG; PKM taxonomies — Zettelkasten, PARA, Johnny.Decimal) converged on a few
transferable principles:
- **Separate raw from synthesized, and keep raw immutable.** Karpathy's core
invariant: interpret and correct in a synthesized layer that *cites* the raw
source; never edit the source. Gives every claim an auditable provenance chain.
- **The index is the catalog — don't hand-maintain one.** Karpathy's `index.md`
exists because a filesystem has no ranker. AKM already has one (plus a utility
signal), so effort should go into per-asset *retrievability* (strong title,
accurate first paragraph, tags, xrefs), not a hand-kept catalog.
- **Metadata/facets beat folder depth for retrieval.** The literature repeatedly
measures metadata-enriched and "contextual"/self-situating retrieval as
materially better (e.g. contextual headers cutting failed retrievals by a large
margin) — because the reader is a ranker, not a browser.
- **Atomicity + sparse, real links (Zettelkasten), not dense mandatory links.**
Backlinks help multi-hop recall, but forced density produces junk edges.
- **Controlled vocabulary prevents bucket drift**, but deep hierarchies and
opaque codes (Johnny.Decimal) fail when nobody browses.
## The debate
Four positions were argued and adversarially critiqued:
| Position | Thesis | Verdict |
|---|---|---|
| **Domain-first taxonomy** | Path = stable subject domain; lowest rename rate, highest lexical signal. | reject-keep-ideas |
| **Scope-first namespacing** | Path = project/client/team; scope is the dominant retrieval filter. | reject-keep-ideas |
| **Flat + faceted + dense backlinks** | Minimal folders; organizing signal lives in frontmatter + a dense link graph the index can read. | adopt-with-changes |
| **Hybrid "narrowest stable scope" ladder** | One folder = the narrowest stable boundary (client > project > domain > root), rest in facets. | adopt-with-changes |
The critiques were decisive:
- **A single global axis fails both ways.** Domain-first makes project scope
invisible (no `project` FTS field; agents that write `project:` frontmatter get
zero retrieval benefit) and splits "everything about project A's auth" across
three query idioms. Scope-first strands reusable assets behind an arbitrary
project home and creates a `shared/` dumping ground.
- **The per-asset "narrowest scope" ladder is non-deterministic.** Two agents run
the delete/archive test on equivalent insights and pick different rungs,
fragmenting a topic across `memory:projectA/` and `knowledge/auth/` with no
reconciliation pass — and the wrong guess is baked into an immutable ref.
- **Mandatory dense xrefs actively hurt.** Because ref strings fold into the FTS
`hints` field, a link quota smears a popular source's ref token across dozens of
citers (destroying its ranking signal) and fills the entity graph with
plausible-but-wrong edges.
- **Per-namespace hub wikis rot.** As an obligation they contend on concurrent
writes, generate stale-index lint noise, and flatten the multi-hop graph into a
namespace-wide star.
## Decision
**Resolve the project-vs-domain tension by asset TYPE, not per-asset judgment.**
This is the one rule that is both deterministic mid-task and mechanically
justified:
- **Scope-born types** (`memory`, `lesson`, `task`, `env`, `secret`) → the
current **project/client** slug. They are born bound to the work in front of
the agent, so the working context *is* the answer — zero counterfactual
reasoning, and `akm search "projectA" --type memory` reconstructs the project.
- **Reuse-born types** (`knowledge`, `skill`, `wiki`, `fact`, `script`) → a
stable **domain** from a short closed vocabulary. Project relevance is already
recovered by scoped-utility ranking (fact 5), so spending the path on the
project axis is redundant; the domain prefix is the only handle that
co-locates cross-project reuse. For a wiki, the domain names the wiki itself.
- **Global-by-nature types** (`command`, `agent`, `workflow`, bundle-wide `env`) →
type root or a tool slug.
Supporting rules, all shipped as convention facts:
- One path axis; depth 1 (2 max for strict containment); lowercase-hyphen
semantic segments; **no volatile facets in the path** (status/date/version/…);
flat type-root fallback instead of inventing a one-off domain.
- Off-axis facets go in `tags` (never a bare `project:` field); the path
scope/domain token merges into `tags` automatically, so it never needs
restating (fact 6).
- **A provenance xref is mandatory when the asset derives from another** (never
invented for original observations). Associative xrefs are discretionary and
capped (~5, a heuristic). Corrections are new assets that xref what they
correct, paired with `beliefState: superseded`/`supersededBy` on the stale
asset; immutability applies to ingested material (wiki `raw/`, vendored docs,
transcripts) — synthesized knowledge pages are the rewritable layer (Karpathy
immutability inside AKM's flat model).
- **Self-situating text on every asset** — the orientation line goes in
`description:`/`when_to_use:` (the higher-weight indexed fields; bounded
body prose also enters the lowest-weight `content` field). The body header feeds the
entity graph (extracted from body prose) and human readers at `akm show`.
Anthropic's Contextual Retrieval figures (35%/49%/67% top-20 failure-rate
reduction) were measured on chunk-prepended context in a chunked RAG pipeline —
an analogy motivating the description-field placement, not a measurement of it.
- Hubs are optional wiki assets for a few high-traffic domains, never a per-write
obligation. The FTS index is the catalog.
- A ref is an address chosen once: **default to not renaming**; if unavoidable,
move the file, grep the bundle for the old ref and fix inbound xrefs in the
same pass, then `akm index` and `akm lint`. The destination is a new identity
and starts with fresh learned state.
Rejected over-builds: mandatory dense/bidirectional xrefs, per-namespace hub
wikis, the per-asset scope ladder, and any hand-maintained catalog.
## What shipped
Convention facts in the bundle skeleton (`src/assets/stash-skeleton/facts/conventions/`):
- `organization.md` (`category: convention`) — the partition-by-type placement
rules. Surfaced to every non-wiki author.
- `backlinks.md` (`category: convention`) — the cross-linking / provenance /
canonical-naming rules. Surfaced to every non-wiki author.
- `domains.md` (`category: convention`) — the editable domain vocabulary and
canonical entity spellings for reuse-born assets.
- A short **Placement & linking** section added to each
`facts/conventions/assets/<type>.md`, stating that type's default axis
(surfaced type-scoped).
Because these are `category: convention` facts, `resolveStashStandards` injects
them into the authoring context automatically — no code change was required to
enforce the surfacing. They are soft guidance; a bundle owner edits them to match
how their bundle is queried.
The code changes the finalized conventions imply are specced separately in
[stash-conventions-code-spec.md](stash-conventions-code-spec.md) (8 specs,
prioritized and sequenced; nothing implemented in this change).
## Review round
A two-reviewer adversarial review (retrieval mechanics / IR + KM methodology /
agent usability), with cross-rebuttals, tuned the shipped conventions. Decisive
corrections, each code-verified or empirically tested:
- **The ref-prefix search idiom was false.** `akm search "<type>:<prefix>/"`
never worked (`sanitizeFtsQuery` strips `:` and `/`; `entry_type` is not an
FTS column) — live-tested at zero hits. All convention text now uses
`akm search "<slug>" --type <type>`. *Amendment (2026-07-11): SPEC-4 has
since made the idiom real — `<type>:<prefix>/` (and bare `<type>:`) queries
now translate to a typed subtree enumeration (`parseRefPrefixQuery`). The
finding was correct at review time; the shipped convention facts keep the
`--type` idiom for now (re-adoption deferred one release — see open
questions).*
- **"Folders are invisible to the ranker" was backwards.** Subpath tokens are
indexed in the FTS `name` column at the highest weight and earn the cwd
project-context boost; the claimed "indexing-confidence bump" does not exist.
- **xrefs never feed the entity graph** — extraction reads body prose
(memory + knowledge) with frontmatter stripped, so graph-relevant
relationships must be stated in the body.
- **Body prose is not FTS/embedding-indexed** — self-situating orientation
belongs in `description:`/`when_to_use:`; the body header serves the graph
and human readers.
- **"Rename-proof" was misleading** — a rename orphans the asset's utility
history (new `entry_key` row), strengthening the no-rename rule.
*Amendment (2026-07-12): SPEC-7 (`akm mv`) made a tooled rename preserve that
history.* **Superseded (0.9.0): a rename is delete plus create and orphans
the utility history. `akm mv`'s in-place re-key still ships, but it is
Experimental and nothing may be built on it, so the original finding stands
unqualified.**
- **Corrections now pair with `beliefState: superseded`/`supersededBy`**
(parsed for all markdown types, demoted at rank time), and immutability was
rescoped to ingested material only, resolving a contradiction with the
knowledge type conventions.
- Completeness: `script` joined the reuse-born bucket; wikis take the domain as
the wiki NAME; `env` got a decision test; a `conventions` domain entered the
vocabulary (the shipped facts had violated their own list); a two-domain
tie-break was added; and duplicated pointer text was trimmed from the nine
co-injected per-type files.
## Open questions / future work
Items with an implementation path were specced in
[stash-conventions-code-spec.md](stash-conventions-code-spec.md), and all
eight specs have since landed (amended 2026-07-12): the lint frontmatter
channels (SPEC-1), the tag merge (SPEC-2), `--xref` (SPEC-3), the ref-prefix
filter (SPEC-4), `--supersedes` demotion (SPEC-5), category capture (SPEC-6 —
shipped capture-only; the rank-time demotion was dropped after measurement
showed no crowding), `akm mv` (SPEC-7), and bounded native body indexing
(SPEC-8) — see the closed bullets below. What remains genuinely open:
vocabulary governance, scheduled consolidation (argued down in the spec — the
improve pipeline already injects the amended conventions), and the typed
provenance channel.
- **A thin HARD floor.** `akm lint` runs a deterministic `missing-ref` check
over body text and `refs:` frontmatter for non-wiki markdown assets; the gap
was precisely the `xrefs:`/`supersededBy:`/`contradictedBy:` frontmatter
channels the conventions mandate — closed by SPEC-1 in
[stash-conventions-code-spec.md](stash-conventions-code-spec.md) (implemented:
the same check now scans those keys too). Orphan and uncited-source checks for
non-wiki assets are argued down (orphanhood is the sanctioned normal state
under discretionary linking; uncited-source is undecidable without knowing
whether an asset is derived).
- **Kill the tag footgun in code.** Closed by SPEC-2 in
[stash-conventions-code-spec.md](stash-conventions-code-spec.md)
(implemented: directory tokens from the canonical ref subpath now always
merge into `tags`; filename tokens still derive only when `tags` is empty).
Takes effect on reindex; operators with collapse-detector canary sets should
re-mint baselines via `akm improve canary --refresh`.
- **Vocabulary governance.** How an agent proposes a `fact:conventions/domains`
addition mid-task without blocking on human review (interim: flat type-root
fallback + a proposal note).
- **Scheduled consolidation.** A curate/consolidate trigger to dedup near-duplicate
atoms, promote recurring memories into domain knowledge, and fix danglers —
since no reorg pass happens organically and utility ranking is cold-start-biased
toward incumbents.
- **Naming collision.** The convention's "scope" wording vs the existing
`entry.scope` (user/agent/run/channel) / `scopeKey` concepts in code — the
convention facts deliberately say "project/client slug" and "partition axis" to
avoid cross-wiring; keep it that way.
- **Ref-prefix filter in code.** Closed by SPEC-4 in
[stash-conventions-code-spec.md](stash-conventions-code-spec.md)
(implemented: `parseRefPrefixQuery` detects `<type>:<prefix>/` — and bare
`<type>:` — queries on the untyped search path and translates them to a
typed index enumeration with name-prefix narrowing; hits carry the fixed
browse score 1, an explicit `--type` wins over the parsed type, and the
in-memory prefix filter leaves an `entry_key LIKE` SQL push-down as a later
optimization). Whether the skeleton convention facts should re-adopt the
idiom over `akm search "<slug>" --type <type>` stays deferred one release so
older CLI versions aren't taught a query shape they don't support.
- **A tooled rename.** Closed by SPEC-7, then **re-opened in 0.9.0 and answered
"not as a contract"**: `akm mv` ships Experimental and a rename is delete plus
create. The recommended forced-rename procedure is the manual one this
document originally prescribed — move the file (a memory's `.derived.md` twin
moves with it), grep and fix inbound refs in the same pass, then `akm index`
and `akm lint`. The no-rename default is again the primary defence, which
strengthens it. See
[0.9.0-decisions.md D3](0.9.0-decisions.md#d3--renames-are-delete--create-akm-mv-ships-experimental).
- **Index self-situating body text.** Closed by SPEC-8 in
[stash-conventions-code-spec.md](stash-conventions-code-spec.md)
(implemented: the native adapter normalizes bounded Markdown prose into the
lowest-weight `content` field; secrets, env files, raw sessions, and session
checkpoints are excluded at the adapter boundary). The
`description:`/`when_to_use:` orientation routing the conventions teach
remains primary.
- **Typed provenance channel.** Evaluate `sources:` for all types, indexed
outside `hints` — it would separate provenance from associative links and
resolve the xref-cap tension mechanically. Rejected for the shipped
convention because non-wiki `sources:` earns no retrieval benefit today and
provenance-in-xrefs powers the "what did we build on this?" query.
- **Convention facts crowd domain-term queries.** Their name/description/
when_to_use tokens (bodies are unindexed) match via prefix expansion and carry
the fact type boost (0.22); consider excluding `category: convention` facts
from default untyped search (their delivery channel is prompt injection,
parallel to the `session` default exclusion) or demoting the category at rank
time. *Amendment (2026-07-12): closed by SPEC-6 in
[stash-conventions-code-spec.md](stash-conventions-code-spec.md), shipped
capture-only. The `category:` key is now captured into the index as
`entry.category` (takes effect on reindex), making a category-keyed policy
implementable — but the spec's prescribed measurement (full skeleton
convention facts plus a real `knowledge/auth` asset, untyped `auth` query,
semantic off) showed NO crowding: FTS is exact-first, so prefix expansion
onto the facts' tokens fires only when nothing matches the query exactly,
and a real domain asset always outranks the facts. The rank-time demotion
contributor was therefore dropped;
`tests/integration/search-convention-fact-demotion.test.ts` pins the no-crowding
invariant and is the regression guard if demotion is ever revisited.*
- **Automate correction demotion.** Closed by SPEC-5 in
[stash-conventions-code-spec.md](stash-conventions-code-spec.md)
(implemented as the deterministic CLI form: `--supersedes <old ref>` on
`akm remember`/`akm import` writes the correction with the xref, sets
`beliefState: superseded` + `supersededBy: [<new ref>]` on the old asset,
and reindexes it; demoting belief states now also cap the final search
score so the stale incumbent cannot outrank its correction). An LLM-side
`curate`/`improve` pass that detects citing corrections automatically
remains open.