UNPKG

@tanstack/ai-sandbox

Version:

Provider-agnostic sandbox layer for TanStack AI — run harness adapters inside isolated sandboxes (defineSandbox, defineWorkspace, withSandbox) with a uniform SandboxHandle, workspace bootstrap, policy, and resumable lifecycle.

1,216 lines 50.3 kB
/**
 * Provider conformance for the two unattended sweeps: `pruneJournals`
 * (`journal-sweep.ts`) and `reapDetachedRuns` (`reap.ts`), against a REAL
 * sandbox.
 *
 * WHY THIS EXISTS SEPARATELY FROM THE UNIT TESTS. Both sweeps are almost
 * entirely *shell* — `ls -1`, `stat -c '%Y %n'`, `rm -f`, `tail -c -N | base64`
 * — composed as strings by `journal.ts` and executed by a provider. The unit
 * suites drive fakes: an `exec` that answers from a scripted table, a
 * filesystem that is a `Map`. A fake cannot be wrong about `stat` the way a
 * BusyBox actually is, and on this feature that gap has already produced four
 * defects that every unit test passed (see `takeover-conformance.ts`'s module
 * doc for the roster). So the four properties the sweeps rest on are asserted
 * here through a real shell against real files:
 *
 * 1. **A deletion really deletes, and a keep really keeps.** Asserted with
 *    `test -f` through the provider's shell, NEVER `handle.fs.exists`: on
 *    local-process the two resolve `/tmp` differently, so an `fs` probe answers
 *    about a path the journal was never written to (`journal.ts` rule 3). A
 *    sweep that "succeeded" while deleting nothing passes an `fs` probe.
 * 2. **The age gate's self-witness works on THIS shell.** `journalMtimeListCommand`
 *    passes the directory as `stat`'s own first operand precisely because
 *    BusyBox exits 1 with EMPTY stdout on an unrecognised flag, and an empty
 *    parse read as an empty directory would delete every live run's journal. The
 *    docker provider's image is `alpine:3` — BusyBox 1.37, where `find -newermt`
 *    and `find -printf` are unrecognised — so the docker matrix is the authority
 *    on this case, not the local-process one (on Windows local-process execs
 *    through git-bash, whose `find`/`stat` are GNU-flavoured).
 * 3. **The reaper never drives a live run.** The `'producing'` case asserts
 *    ABSENCE — nothing appended, `close()` not called, not one `runs.update`,
 *    `detachedSince` intact — because that is the shape of the defect
 *    `probeRunExit` exists to prevent: entering `pipeToRunLog` to "check" writes
 *    a terminal status and drops the run out of `listReclaimable` forever.
 * 4. **A shell-hostile runId cannot become a shell-hostile command.** The encode
 *    → journal → follow → `ls` → decode → `rm` round trip runs on a runId
 *    containing `/`, a space, `;`, `$( )` and an embedded `touch`, with a canary
 *    file asserted absent. An ENCODING bug here is arbitrary command execution
 *    inside the sandbox, not a cosmetic defect.
 *
 *    **What the canary proves, exactly, and what it does not.** It detects a
 *    runId reaching the shell WITHOUT `encodeRunId` — that is the mutation it
 *    bites on, and it bites hard: `journaledCommand`, `journalFollowCommand`,
 *    `journalExitProbeCommand`, `journalStderrReadCommand` and
 *    `journalCleanupCommand` all interpolate the path, so the `;touch` executes
 *    and the canary appears. It is BLIND to the loss of `journal.ts`'s
 *    `shellQuote`, the second and independent layer. Measured: with `shellQuote`
 *    reduced to the identity while `encodeRunId` stays, the redirect target
 *    becomes `>> /tmp/…/rp-a_3btouch_20_2ftmp…ndjson` — a single shell word of
 *    `[A-Za-z0-9._/-]`, because the encoder already removed every character a
 *    shell can act on — so no canary fires and NOTHING in this suite, or in any
 *    other real-provider suite, changes. Do not read a green run here as licence
 *    to "simplify" `shellQuote` away.
 *
 *    The quoting is pinned instead by exact-string unit tests in
 *    `packages/ai-sandbox/tests/journal.test.ts`, which compare each composed
 *    command to a literal containing the quotes. By name, one per command:
 *    `journaledCommand` — "redirects stdout to the journal, stderr to its own
 *    file, and appends the exit sentinel" plus "quotes an adversarial runId so it
 *    cannot inject shell metacharacters"; `journalFollowCommand` — "translates a
 *    0-based consumed-byte count into tail -c +N (1-based)";
 *    `journalReadCommand` — "the bounded read drops -f and keeps the base64
 *    frame, so a poll cannot hang"; `journalExistsCommand` — "probes through the
 *    shell, never through fs.*"; `journalStderrReadCommand` — "reads a BOUNDED
 *    tail of the sidecar, base64-framed, stderr silenced";
 *    `journalCleanupCommand`, `journalMtimeListCommand` and
 *    `journalExitProbeCommand` — the first `it` under each of their `describe`s.
 *    Those are the tests that go red on a dropped `shellQuote`; keep them exact.
 *
 * A provider that cannot satisfy the contract MUST declare `unsupported.reason`.
 * As in the journal and takeover suites there is deliberately no silent-skip
 * path: a conformance case that quietly returns prints as a pass, which is how
 * an unimplemented capability ships green.
 *
 * EVERY WAIT IN THIS FILE IS BOUNDED, and every journal directory is unique per
 * case — see {@link caseDir}. This suite DELETES FILES, and
 * `DEFAULT_JOURNAL_DIR` is a fixed absolute path shared with every other test
 * and, on local-process, with a developer's real runs.
 *
 * Vitest is an OPTIONAL peer dependency: this module is imported only from test
 * files, which already run under Vitest.
 */
import { randomUUID } from 'node:crypto'
import { describe, expect, it } from 'vitest'
import { EventType, InMemoryRunStore } from '@tanstack/ai'
import { InMemoryLockStore } from '@tanstack/ai/locks'
import {
  EXIT_SENTINEL_KEY,
  decodeJournalRunId,
  exitSentinelLine,
  journalCleanupCommand,
  journalExistsCommand,
  journalListCommand,
  journalMtimeListCommand,
  journalPaths,
  journalReadCommand,
  journalStderrReadCommand,
  journaledCommand,
  parseJournalMtimeListing,
} from '../journal'
import { journalReadStrategy, readJournal } from '../journal-reader'
import { pruneJournals } from '../journal-sweep'
import { probeRunExit, reapDetachedRuns } from '../reap'
import { readJournalNdjson } from '../runner'
import { chunkFingerprint, createRunScopedIdGen } from '../chunk-identity'
import { waitForJournal } from './journal-conformance'
import type { JournalPaths } from '../journal'
import type { SandboxHandle } from '../contracts'
import type { RunStore, StreamChunk, StreamDurability } from '@tanstack/ai'

export interface ReaperConformanceConfig {
  /** Provider name, used in the describe title. */
  name: string
  /** Create a live sandbox plus its teardown. */
  createHandle: () => Promise<{
    handle: SandboxHandle
    dispose: () => Promise<void>
  }>
  /**
   * Declare that this provider cannot support the sweeps, with the reason.
   * Registers a skipped case whose title carries the reason — a NAMED skip,
   * visible in the reporter. Omit it and the suite runs.
   */
  unsupported?: { reason: string }
  /**
   * Declare that this provider's reads take the POLL strategy rather than the
   * FOLLOW one — i.e. `journalReadStrategy` answers `'poll'` for its handles.
   *
   * Only the FOLLOW half of the shell-hostile-runId case depends on it, so this
   * does not skip a case; it names itself in that case's title and the follow
   * read is omitted. The declaration is checked against the live handle there, in
   * both directions, so it cannot quietly remove coverage from a provider that
   * can in fact follow.
   */
  followUnsupported?: { reason: string }
  /**
   * Declare that this provider cannot run GNU `stat -c '%Y %n'`. The three
   * age-gate cases skip with this reason. Docker alpine is the authority on
   * the witness line; local-process on Darwin is BSD `stat`.
   */
  mtimeListUnsupported?: { reason: string }
}

/** Poll interval handed to providers that cannot follow a growing file. */
const POLL_INTERVAL_MS = 50

/**
 * Quiescence window for the reaper's first append. Short because the agent in
 * these cases has provably stopped (the suite waited for its sentinel) — the
 * gate still runs, it just does not need to wait 5s to observe nothing.
 */
const FENCE_QUIET_MS = 25

/**
 * Bound on a real journal read, so a reader that delivers nothing FAILS instead
 * of parking CI.
 *
 * Never an assertion, and deliberately far above anything a healthy read needs
 * (measured: 10–18s for the follow cases on both providers). Every use site
 * pairs it with a `backstopped: false` witness, so a read the CLOCK ended fails
 * naming this backstop rather than as a downstream transcript mismatch — which
 * means this number can be raised freely and must never be the thing a case is
 * tuned against.
 */
const READ_BACKSTOP_MS = 90_000

/** Long enough that nothing in this suite is ever classified as expired. */
const NEVER_EXPIRES_MS = 60 * 60 * 1000

/**
 * A journal directory nothing else on the machine writes to, created fresh for
 * EVERY case.
 *
 * Not `DEFAULT_JOURNAL_DIR`, and not even one directory per suite. Both sweeps
 * under test enumerate a whole directory and then DELETE from it, so a shared
 * directory would let one case's leftovers become another's input — and on
 * local-process the sandbox shell shares the host's real `/tmp`, where
 * `DEFAULT_JOURNAL_DIR` holds a developer's actual runs.
 */
function caseDir(): string {
  return `/tmp/tanstack-reaper-conformance-${randomUUID()}`
}

/**
 * Unique per run, and it must be: `journalPaths` derives the filename from the
 * runId and the journal is append-only, so a reused id appends BEHIND the
 * previous run's `{"__exit":N}` sentinel and the new run appears to emit nothing
 * at all (see `journal.ts`).
 */
function uniqueRunId(label: string): string {
  return `rp-${label}-${randomUUID()}`
}

/**
 * Single-quote a shell word, POSIX-style — the same rule `journal.ts`'s private
 * `shellQuote` applies.
 *
 * Duplicated rather than exported from production code on purpose: this exists
 * only for this suite's `rm -rf` teardown, which is not a production operation
 * and must not become one by growing an export for it.
 */
function quote(value: string): string {
  return `'${value.replaceAll("'", `'\\''`)}'`
}

/** Remove a case's journal directory and everything in it. Best effort. */
async function removeDir(handle: SandboxHandle, dir: string): Promise<void> {
  try {
    await handle.process.exec(`rm -rf ${quote(dir)}`)
  } catch {
    // The sandbox may already be gone, and on docker it is about to be. Nothing
    // under test depends on the directory being absent afterwards — the cases
    // that DO assert deletion assert it directly, per file.
  }
}

/**
 * Does `path` exist, according to the SANDBOX'S SHELL?
 *
 * `journalExistsCommand` rather than `handle.fs.exists`, for any path and not
 * just a journal: `journal.ts` rule 3 — on local-process `fs.*` resolves `/tmp`
 * under the sandbox root while a shell redirect hits the host's real `/tmp`, so
 * an `fs` probe would answer about a different file and every deletion
 * assertion in this suite would pass vacuously.
 */
async function fileExists(
  handle: SandboxHandle,
  path: string,
): Promise<boolean> {
  const probe = await handle.process.exec(
    // Only `journal` is read by the probe, and its parameter is typed
    // `Pick<JournalPaths, 'journal'>` for exactly this reason: an arbitrary path
    // has no run behind it, so there is no nonce or sidecar to invent.
    journalExistsCommand({ journal: path }),
  )
  return probe.exitCode === 0
}

/** Filename as `ls -1` reports it, for a path inside `dir`. */
function basename(dir: string, path: string): string {
  return path.slice(dir.length + 1)
}

/**
 * A real agent: a shell command printing one NDJSON line per delta, then
 * exiting.
 *
 * `printf '%s\n' a b c` reuses the format for every operand on GNU coreutils
 * and on BusyBox alike, so this needs no loop. The JSON contains only double
 * quotes, so it is safe inside the POSIX single-quoted words this builds.
 */
function emitLines(deltas: Array<string>): string {
  return `printf '%s\\n' ${deltas.map((delta) => `'{"delta":"${delta}"}'`).join(' ')}`
}

/**
 * Run a journaled agent to completion, so the `{"__exit":N}` sentinel is in the
 * journal by the time this resolves.
 *
 * `exec`, not `spawn`: `exec` waits, and a bounded wait is the only kind this
 * suite allows. (`SpawnHandle.wait()` is also not safe to call after the fact on
 * every provider — see `journal-conformance.ts`.)
 */
async function runAgent(
  handle: SandboxHandle,
  paths: JournalPaths,
  deltas: Array<string>,
): Promise<void> {
  await handle.process.exec(journaledCommand(emitLines(deltas), paths))
}

/** `ls -1` output as a list of names. */
async function listNames(
  handle: SandboxHandle,
  dir: string,
): Promise<Array<string>> {
  const listing = await handle.process.exec(journalListCommand(dir))
  return listing.stdout
    .split('\n')
    .map((line) => line.trim())
    .filter((line) => line !== '')
}

/** Decode the base64 frame a bounded journal read produces. */
function decodeJournalRead(stdout: string): string {
  return Buffer.from(stdout.replace(/\s+/g, ''), 'base64').toString('utf8')
}

/**
 * The `stat -c '%Y %n'` listing for `dir`, plus the raw stdout so a case can
 * assert the WITNESS LINE itself rather than only its parsed consequence.
 */
async function mtimeListing(
  handle: SandboxHandle,
  dir: string,
): Promise<{ stdout: string; entries: Map<string, number> }> {
  const probe = await handle.process.exec(journalMtimeListCommand(dir))
  const parsed = parseJournalMtimeListing(probe.stdout, dir)
  if (parsed.kind !== 'listed') {
    throw new Error(
      `reaper conformance: the mtime listing for ${dir} came back unavailable — ` +
        `stat -c '%Y %n' produced no witness line. stdout: ${JSON.stringify(probe.stdout)}`,
    )
  }
  return {
    stdout: probe.stdout,
    entries: new Map(
      parsed.entries.map((entry) => [entry.name, entry.mtimeMs]),
    ),
  }
}

/**
 * Is there a `<seconds> <dir>` line — `stat`'s report on its own first operand?
 *
 * That line, not the exit status, is the evidence the mechanism ran: BusyBox
 * exits 1 both for an EMPTY directory (whose unexpanded glob it cannot stat) and
 * for an unrecognised flag, and only the witness distinguishes them.
 */
function hasWitnessLine(stdout: string, dir: string): boolean {
  return stdout
    .split('\n')
    .some((line) => /^\d+ (?<path>.+)$/.exec(line.trim())?.[1] === dir)
}

/** Read one file's mtime out of a listing, loudly when it is missing. */
function mtimeOf(entries: Map<string, number>, name: string): number {
  const mtimeMs = entries.get(name)
  if (mtimeMs === undefined) {
    throw new Error(
      `reaper conformance: ${name} has no mtime in the stat listing, so the age gate cannot be exercised`,
    )
  }
  return mtimeMs
}

/**
 * An in-process event log with real accumulated state, plus the two facts the
 * reaper assertions need: what was appended, and how many times `close()` ran.
 *
 * `close()` is the load-bearing counter. `pipeToRunLog` ALWAYS calls it, so a
 * reaper that entered the pipe to find out whether a run finished would show up
 * here as `closes() === 1` — which ends every attached client's stream — even if
 * it happened to append nothing.
 */
interface ConformanceLog {
  log: StreamDurability
  stored: () => Array<StreamChunk>
  closes: () => number
}

function conformanceLog(): ConformanceLog {
  const entries: Array<{ offset: string; chunk: StreamChunk }> = []
  let closes = 0
  return {
    log: {
      resumeFrom: () => null,
      append: (chunks) =>
        Promise.resolve(
          chunks.map((chunk) => {
            const offset = `reap:${entries.length}`
            entries.push({ offset, chunk })
            return offset
          }),
        ),
      // Nothing here tails the log — every assertion reads the appended
      // transcript, and a `read` would park until `close()` (see `align.ts`).
      read: () => (async function* empty() {})(),
      close: () => {
        closes += 1
        return Promise.resolve()
      },
      snapshot: () => Promise.resolve(entries.map((entry) => ({ ...entry }))),
    },
    stored: () => entries.map((entry) => entry.chunk),
    closes: () => closes,
  }
}

/**
 * A `RunStore` that counts its MUTATIONS, so the leave-alone case can assert
 * that a producing run's record was not written at all.
 *
 * "Status still `'running'`" is too weak on its own: `driverEpoch` is bumped by
 * `withRunClaim` before any status is written, so a reaper that claimed a live
 * run and then bailed would still read as `'running'`. Counting `update` sees
 * that; reading the status does not.
 */
interface CountingRunStore {
  runs: RunStore
  updates: () => number
}

function countingRunStore(inner: InMemoryRunStore): CountingRunStore {
  let updates = 0
  return {
    runs: {
      createOrResume: (...args) => inner.createOrResume(...args),
      update: (...args) => {
        updates += 1
        return inner.update(...args)
      },
      get: (...args) => inner.get(...args),
      listByThread: (...args) => inner.listByThread(...args),
      listReclaimable: (...args) => inner.listReclaimable(...args),
      findActiveRun: (...args) => inner.findActiveRun(...args),
    },
    updates: () => updates,
  }
}

/** The event a journal line translates into. `timestamp` is excluded from `chunkFingerprint`. */
function contentChunk(messageId: string, delta: string): StreamChunk {
  return {
    type: EventType.TEXT_MESSAGE_CONTENT,
    messageId,
    delta,
    timestamp: Date.now(),
  }
}

/**
 * Narrow one parsed journal line into its chunk.
 *
 * Fields are validated and the chunk REBUILT from them rather than asserted into
 * shape: a cast would let a provider that mangles the bytes reach
 * `chunkFingerprint` as a structurally invalid chunk and fail somewhere
 * unrelated.
 */
function toChunk(
  runId: string,
  messageId: string,
  value: unknown,
): StreamChunk {
  if (typeof value !== 'object' || value === null || !('delta' in value)) {
    throw new Error(
      `reaper conformance: run ${runId} journal line is not an agent event: ${JSON.stringify(value)}`,
    )
  }
  const delta = value.delta
  if (typeof delta !== 'string') {
    throw new Error(
      `reaper conformance: run ${runId} journal line has a non-string delta: ${JSON.stringify(value)}`,
    )
  }
  return contentChunk(messageId, delta)
}

/** Deterministic translator: re-reading the journal reproduces the same chunks. */
async function* translate(
  runId: string,
  lines: AsyncIterable<unknown>,
): AsyncIterable<StreamChunk> {
  const messageId = createRunScopedIdGen(runId)()
  for await (const line of lines) yield toChunk(runId, messageId, line)
}

/** A comparable transcript: each chunk reduced to its fingerprint. */
function transcript(chunks: Array<StreamChunk>): Array<string> {
  return chunks.map(chunkFingerprint)
}

/** The chunks a run over `deltas` must deliver, exactly once and in order. */
function expectedTranscript(
  runId: string,
  deltas: Array<string>,
): Array<StreamChunk> {
  const messageId = createRunScopedIdGen(runId)()
  return deltas.map((delta) => contentChunk(messageId, delta))
}

/**
 * The reaper's `drive`: read the run's journal from byte 0 and translate it.
 *
 * The read is bounded independently of `signal` so a journal that stops growing
 * fails the case instead of hanging CI.
 *
 * Returns the drive alongside `backstopped()`, the causal witness for
 * {@link READ_BACKSTOP_MS}: the case must assert it is `false` before its
 * transcript assertions, so a read the CLOCK ended fails naming the backstop
 * instead of as a truncated-transcript diff.
 */
function driveFromJournal(
  handle: SandboxHandle,
  dir: string,
): {
  drive: (input: {
    runId: string
    threadId: string
    signal: AbortSignal
  }) => AsyncIterable<StreamChunk>
  /** True if any read this drive started was ended by the backstop clock. */
  backstopped: () => boolean
} {
  // One entry per `drive` invocation, so a sweep that drives more than one run
  // cannot hide a backstopped read behind a healthy one.
  const backstops: Array<AbortSignal> = []
  return {
    drive: ({ runId, signal }) => {
      // Not the assertion — see {@link READ_BACKSTOP_MS}. `backstopped()` is what
      // proves the clock was not what ended the read.
      const backstop = AbortSignal.timeout(READ_BACKSTOP_MS)
      backstops.push(backstop)
      return translate(
        runId,
        readJournalNdjson(handle, {
          signal: AbortSignal.any([signal, backstop]),
          journal: { runId, dir, pollIntervalMs: POLL_INTERVAL_MS },
        }),
      )
    },
    backstopped: () => backstops.some((s) => s.aborted),
  }
}

/** A `'running'`, DETACHED record — the shape `listReclaimable` selects on. */
async function detachedRun(
  store: RunStore,
  runId: string,
  threadId: string,
  detachedSince: number,
): Promise<void> {
  await store.createOrResume({ runId, threadId, startedAt: Date.now() })
  await store.update(runId, { detachedSince })
}

function sleep(ms: number): Promise<void> {
  return new Promise((resolve) => setTimeout(resolve, ms))
}

/**
 * Assert `createHandle` satisfies the sweep conformance contract. Each `it` gets
 * a fresh sandbox via `createHandle`/`dispose`, a fresh journal directory, and
 * unique runIds, so no case can observe another's files.
 */
export function runReaperConformance(config: ReaperConformanceConfig): void {
  describe(`reaper conformance — ${config.name}`, () => {
    if (config.unsupported) {
      it.skip(`unsupported: ${config.unsupported.reason}`, () => {
        expect(true).toBe(true)
      })
      return
    }

    // -----------------------------------------------------------------------
    // 1. `pruneJournals` against a real filesystem.
    // -----------------------------------------------------------------------
    it(
      "deletes a terminal run's journal AND its .err sidecar, while a running run's journal survives the same sweep",
      { timeout: 60_000 },
      async () => {
        const { handle, dispose } = await config.createHandle()
        const dir = caseDir()
        const terminalId = uniqueRunId('terminal')
        const liveId = uniqueRunId('live')
        const terminal = journalPaths(terminalId, dir)
        const live = journalPaths(liveId, dir)
        try {
          await runAgent(handle, terminal, ['1'])
          await runAgent(handle, live, ['1'])
          // Premise: all four files really exist before the sweep, otherwise
          // "deleted" below would be indistinguishable from "never written".
          expect({
            terminalJournal: await fileExists(handle, terminal.journal),
            terminalSidecar: await fileExists(handle, terminal.stderr),
            liveJournal: await fileExists(handle, live.journal),
          }).toEqual({
            terminalJournal: true,
            terminalSidecar: true,
            liveJournal: true,
          })

          const runs = new InMemoryRunStore()
          await runs.createOrResume({
            runId: terminalId,
            threadId: `${terminalId}-t`,
            startedAt: Date.now(),
          })
          await runs.update(terminalId, {
            status: 'completed',
            finishedAt: Date.now(),
          })
          await runs.createOrResume({
            runId: liveId,
            threadId: `${liveId}-t`,
            startedAt: Date.now(),
          })

          const result = await pruneJournals({ handle, runs, dir })
          expect(result.deleted).toEqual([terminalId])
          expect(result.failures).toEqual([])
          expect(result.kept).toEqual([
            { runId: liveId, names: expect.any(Array), reason: 'non-terminal' },
          ])

          // The files, through the shell. A sweep that reported a deletion it did
          // not perform passes every assertion above and fails here.
          expect({
            terminalJournal: await fileExists(handle, terminal.journal),
            terminalSidecar: await fileExists(handle, terminal.stderr),
            liveJournal: await fileExists(handle, live.journal),
            liveSidecar: await fileExists(handle, live.stderr),
          }).toEqual({
            terminalJournal: false,
            terminalSidecar: false,
            liveJournal: true,
            liveSidecar: true,
          })
        } finally {
          await removeDir(handle, dir)
          await dispose()
        }
      },
    )

    it(
      'sweeps the same terminal run twice without a failure, and rm -f of an already-absent journal exits 0',
      { timeout: 60_000 },
      async () => {
        const { handle, dispose } = await config.createHandle()
        const dir = caseDir()
        const runId = uniqueRunId('twice')
        const paths = journalPaths(runId, dir)
        try {
          await runAgent(handle, paths, ['1'])
          const runs = new InMemoryRunStore()
          await runs.createOrResume({
            runId,
            threadId: `${runId}-t`,
            startedAt: Date.now(),
          })
          await runs.update(runId, {
            status: 'completed',
            finishedAt: Date.now(),
          })

          const first = await pruneJournals({ handle, runs, dir })
          expect(first.deleted).toEqual([runId])

          // The second sweep sees an empty directory. It must report nothing to
          // do rather than a failure — a cron runs this every tick forever.
          const second = await pruneJournals({ handle, runs, dir })
          expect({
            listed: second.listed,
            runIds: second.runIds,
            deleted: second.deleted,
            kept: second.kept,
            failures: second.failures,
          }).toEqual({
            listed: 0,
            runIds: 0,
            deleted: [],
            kept: [],
            failures: [],
          })

          // And the `rm -f` the sweep issues is itself idempotent on this shell.
          // Asserted directly because the sweep folds a non-zero `rm` into
          // `kept: 'delete-failed'` and would therefore hide it as a keep.
          const rerun = await handle.process.exec(journalCleanupCommand(paths))
          expect(rerun.exitCode).toBe(0)
        } finally {
          await removeDir(handle, dir)
          await dispose()
        }
      },
    )

    it(
      'leaves a filename it cannot decode alone, while still sweeping the terminal run beside it',
      { timeout: 60_000 },
      async () => {
        const { handle, dispose } = await config.createHandle()
        const dir = caseDir()
        const runId = uniqueRunId('undecodable')
        const paths = journalPaths(runId, dir)
        // `_1.` is not a two-hex-digit escape, so this name is `malformed` — the
        // shape a truncated or foreign file has. `decodeJournalRunId` must refuse
        // it, and the sweep must keep it WITHOUT asking the store, because a
        // plausible-but-wrong runId could answer `terminal` for someone else.
        const strayName = 'reaper-conformance-stray_1.ndjson'
        const strayPath = `${dir}/${strayName}`
        try {
          await runAgent(handle, paths, ['1'])
          await handle.process.exec(
            `printf 'not a journal\\n' >> ${quote(strayPath)}`,
          )
          expect(await fileExists(handle, strayPath)).toBe(true)
          expect(decodeJournalRunId(strayName).kind).toBe('malformed')

          const runs = new InMemoryRunStore()
          await runs.createOrResume({
            runId,
            threadId: `${runId}-t`,
            startedAt: Date.now(),
          })
          await runs.update(runId, {
            status: 'completed',
            finishedAt: Date.now(),
          })

          const result = await pruneJournals({ handle, runs, dir })
          expect(result.deleted).toEqual([runId])
          expect(result.kept).toEqual([
            { names: [strayName], reason: 'undecodable-name' },
          ])
          expect(result.failures).toEqual([])
          expect({
            strayKept: await fileExists(handle, strayPath),
            journalDeleted: !(await fileExists(handle, paths.journal)),
          }).toEqual({ strayKept: true, journalDeleted: true })
        } finally {
          await removeDir(handle, dir)
          await dispose()
        }
      },
    )

    // -----------------------------------------------------------------------
    // 2. The age gate on a real shell.
    // -----------------------------------------------------------------------
    const mtimeSkip = config.mtimeListUnsupported
    const itMtime = (title: string, timeout: number, fn: () => Promise<void>) =>
      it(
        mtimeSkip ? `${title} (unsupported: ${mtimeSkip.reason})` : title,
        { timeout, skip: Boolean(mtimeSkip) },
        fn,
      )

    itMtime(
      "emits stat's self-witness line for a populated directory, so the age gate is usable",
      60_000,
      async () => {
        const { handle, dispose } = await config.createHandle()
        const dir = caseDir()
        const runId = uniqueRunId('witness')
        const paths = journalPaths(runId, dir)
        try {
          await runAgent(handle, paths, ['1'])
          const listing = await mtimeListing(handle, dir)
          // The witness is what makes "no files" distinguishable from "the
          // mechanism is unavailable". On BusyBox 1.37 — the docker provider's
          // `alpine:3` — `find -newermt`/`-printf` are unrecognised and exit 1
          // with empty stdout, which is exactly why the design is a witness line
          // rather than a `find` and an exit code.
          expect(hasWitnessLine(listing.stdout, dir)).toBe(true)
          expect([...listing.entries.keys()].sort()).toEqual(
            [basename(dir, paths.journal), basename(dir, paths.stderr)].sort(),
          )
          // Real epoch times, not the parser's zeroes: a `%Y` the shell did not
          // expand would parse as no entry at all, and a `stat` that printed
          // something else would land far from now.
          for (const mtimeMs of listing.entries.values()) {
            expect(Math.abs(Date.now() - mtimeMs)).toBeLessThan(120_000)
          }
        } finally {
          await removeDir(handle, dir)
          await dispose()
        }
      },
    )

    itMtime(
      'reports an EMPTY journal directory as witness-only rather than unavailable',
      60_000,
      async () => {
        const { handle, dispose } = await config.createHandle()
        const dir = caseDir()
        try {
          await handle.process.exec(`mkdir -p ${quote(dir)}`)
          const listing = await mtimeListing(handle, dir)
          expect(hasWitnessLine(listing.stdout, dir)).toBe(true)
          expect([...listing.entries.keys()]).toEqual([])

          // And the sweep agrees: an empty directory is a LISTED age gate, not an
          // unavailable one. `'unavailable'` here would silently disable orphan
          // expiry forever on this provider.
          const result = await pruneJournals({
            handle,
            runs: new InMemoryRunStore(),
            dir,
          })
          expect({
            listed: result.listed,
            ageGate: result.ageGate,
            deleted: result.deleted,
            failures: result.failures,
          }).toEqual({
            listed: 0,
            ageGate: 'listed',
            deleted: [],
            failures: [],
          })
        } finally {
          await removeDir(handle, dir)
          await dispose()
        }
      },
    )

    itMtime(
      'keeps an orphan younger than orphanTtlMs and sweeps the older one, in the same pass',
      120_000,
      async () => {
        const { handle, dispose } = await config.createHandle()
        const dir = caseDir()
        const olderId = uniqueRunId('older')
        const newerId = uniqueRunId('newer')
        const older = journalPaths(olderId, dir)
        const newer = journalPaths(newerId, dir)
        try {
          await runAgent(handle, older, ['1'])
          // `stat -c '%Y'` is second-granular, so the two runs must be more than
          // one second apart for their ages to be distinguishable at all.
          await sleep(2_500)
          await runAgent(handle, newer, ['1'])

          // The cutoff is computed from the REAL mtimes the real shell reported,
          // not from a fabricated timestamp: that is the whole point of running
          // this against a provider. `pruneJournals` keeps when the NEWEST of a
          // run's files is strictly newer than the cutoff, so placing the cutoff
          // between the two runs must expire exactly one of them.
          const listing = await mtimeListing(handle, dir)
          const newestOf = (paths: JournalPaths): number =>
            Math.max(
              mtimeOf(listing.entries, basename(dir, paths.journal)),
              mtimeOf(listing.entries, basename(dir, paths.stderr)),
            )
          const olderMtime = newestOf(older)
          const newerMtime = newestOf(newer)
          expect(newerMtime - olderMtime).toBeGreaterThanOrEqual(1_000)

          const now = Date.now()
          const cutoff = olderMtime + Math.floor((newerMtime - olderMtime) / 2)
          // NEITHER run is in the store, so both take the orphan arm and only the
          // age gate decides between them.
          const result = await pruneJournals({
            handle,
            runs: new InMemoryRunStore(),
            dir,
            now,
            orphanTtlMs: now - cutoff,
          })
          expect(result.ageGate).toBe('listed')
          expect(result.deleted).toEqual([olderId])
          expect(result.kept).toEqual([
            {
              runId: newerId,
              names: expect.any(Array),
              reason: 'orphan-too-recent',
            },
          ])
          expect(result.failures).toEqual([])
          expect({
            olderJournal: await fileExists(handle, older.journal),
            olderSidecar: await fileExists(handle, older.stderr),
            newerJournal: await fileExists(handle, newer.journal),
            newerSidecar: await fileExists(handle, newer.stderr),
          }).toEqual({
            olderJournal: false,
            olderSidecar: false,
            newerJournal: true,
            newerSidecar: true,
          })
        } finally {
          await removeDir(handle, dir)
          await dispose()
        }
      },
    )

    // -----------------------------------------------------------------------
    // 3. `reapDetachedRuns` end to end.
    // -----------------------------------------------------------------------
    it(
      'finalizes a detached run whose agent reached its sentinel, and its transcript lands',
      { timeout: 120_000 },
      async () => {
        const { handle, dispose } = await config.createHandle()
        const dir = caseDir()
        const runId = uniqueRunId('finalize')
        const threadId = `${runId}-t`
        const deltas = ['1', '2', '3']
        const paths = journalPaths(runId, dir)
        try {
          const runs = new InMemoryRunStore()
          const detachedSince = Date.now()
          await detachedRun(runs, runId, threadId, detachedSince)
          await runAgent(handle, paths, deltas)

          // The probe, on its own, before any sweep: this read is what makes the
          // reaper safe, and it must answer from the JOURNAL rather than from the
          // delivery log (which a detached run's dead host stopped appending to).
          expect(await probeRunExit({ handle, runId, dir })).toEqual({
            state: 'finished',
            exitCode: 0,
          })

          const log = conformanceLog()
          const journalDrive = driveFromJournal(handle, dir)
          const result = await reapDetachedRuns({
            runs,
            locks: new InMemoryLockStore(),
            durability: () => log.log,
            hasFinished: (record) =>
              probeRunExit({ handle, runId: record.runId, dir }),
            drive: journalDrive.drive,
            now: Date.now(),
            detachedRunTtlMs: NEVER_EXPIRES_MS,
            fenceQuietMs: FENCE_QUIET_MS,
          })

          // The causal witness, before anything downstream — see
          // {@link READ_BACKSTOP_MS}. The reaper's read ends at the sentinel; if
          // the clock ended it instead, the transcript below is short and the
          // failure must name the backstop rather than a missing chunk.
          expect({ backstopped: journalDrive.backstopped() }).toEqual({
            backstopped: false,
          })
          expect({
            considered: result.considered,
            probed: result.probed,
            finalized: result.outcomes.finalized,
          }).toEqual({ considered: 1, probed: 1, finalized: 1 })
          expect(result.runs).toEqual([
            { runId, outcome: 'finalized', status: 'completed', exitCode: 0 },
          ])
          // The transcript, element for element — the reaper's whole purpose is
          // that the run a nobody watched still ends up saved.
          expect(transcript(log.stored())).toEqual(
            transcript(expectedTranscript(runId, deltas)),
          )
          const record = await runs.get(runId)
          expect(record?.status).toBe('completed')
          // NEVER CLEARED: `detachedSince` is what the next sweep selects on, and
          // clearing it would reset the TTL on every pass.
          expect(record?.detachedSince).toBe(detachedSince)
        } finally {
          await removeDir(handle, dir)
          await dispose()
        }
      },
    )

    it(
      'reports a still-producing detached run as producing and leaves it completely untouched',
      { timeout: 120_000 },
      async () => {
        const { handle, dispose } = await config.createHandle()
        const dir = caseDir()
        const runId = uniqueRunId('producing')
        const threadId = `${runId}-t`
        const paths = journalPaths(runId, dir)
        const store = countingRunStore(new InMemoryRunStore())
        // A REAL agent that has written a line and is genuinely still alive: no
        // sentinel can be in the journal, and driving it would truncate a healthy
        // run's transcript at line one.
        const agent = await handle.process.spawn(
          journaledCommand(`${emitLines(['1'])}; sleep 30`, paths),
        )
        try {
          const detachedSince = Date.now()
          await detachedRun(store.runs, runId, threadId, detachedSince)
          await waitForJournal(handle, paths)
          const read = await handle.process.exec(journalReadCommand(paths, 0))
          const text = decodeJournalRead(read.stdout)
          // Producing, provably: the first line is there and the sentinel is not.
          expect(text).toContain('{"delta":"1"}')
          expect(text).not.toContain('__exit')
          expect(await probeRunExit({ handle, runId, dir })).toEqual({
            state: 'producing',
          })

          const log = conformanceLog()
          let driveCalled = false
          let durabilityCalls = 0
          const result = await reapDetachedRuns({
            runs: store.runs,
            locks: new InMemoryLockStore(),
            durability: () => {
              durabilityCalls += 1
              return log.log
            },
            hasFinished: (record) =>
              probeRunExit({ handle, runId: record.runId, dir }),
            drive: () => {
              driveCalled = true
              return (async function* never() {})()
            },
            now: Date.now(),
            detachedRunTtlMs: NEVER_EXPIRES_MS,
            fenceQuietMs: FENCE_QUIET_MS,
          })

          expect(result.runs).toEqual([{ runId, outcome: 'producing' }])
          expect({
            considered: result.considered,
            probed: result.probed,
            producing: result.outcomes.producing,
            finalized: result.outcomes.finalized,
            expired: result.outcomes.expired,
            failed: result.outcomes.failed,
          }).toEqual({
            considered: 1,
            probed: 1,
            producing: 1,
            finalized: 0,
            expired: 0,
            failed: 0,
          })

          // ABSENCE, asserted in one object so a regression names which
          // guarantee broke instead of failing on whichever line came first.
          // Every one of these is a way the pre-`probeRunExit` design destroyed a
          // live run: an append duplicates its prefix, a `close()` ends every
          // attached client's stream, an `update` writes `'completed'` and drops
          // the run out of `listReclaimable` forever, and a moved
          // `detachedSince` restarts its TTL.
          const record = await store.runs.get(runId)
          expect({
            driveCalled,
            durabilityCalls,
            appended: log.stored().length,
            closes: log.closes(),
            updatesAfterSetup: store.updates() - 1,
            status: record?.status,
            detachedSince: record?.detachedSince,
            driverEpoch: record?.driverEpoch,
          }).toEqual({
            driveCalled: false,
            durabilityCalls: 0,
            appended: 0,
            closes: 0,
            updatesAfterSetup: 0,
            status: 'running',
            detachedSince,
            driverEpoch: undefined,
          })
        } finally {
          // The agent outlives the sweep on purpose; reap it here so no `sleep`
          // survives the case.
          try {
            await agent.kill()
          } catch {
            // Already gone, or a provider whose sandbox teardown covers it.
          }
          await removeDir(handle, dir)
          await dispose()
        }
      },
    )

    // -----------------------------------------------------------------------
    // 4. A shell-hostile runId, end to end. SECURITY-RELEVANT.
    // -----------------------------------------------------------------------
    it(
      'round-trips a shell-hostile runId through encode, journal, follow, sidecar read, list, decode and delete without executing any of it' +
        (config.followUnsupported === undefined
          ? ''
          : ` (follow read omitted: ${config.followUnsupported.reason})`),
      { timeout: 120_000 },
      async () => {
        const { handle, dispose } = await config.createHandle()
        const dir = caseDir()
        const nonce = randomUUID().slice(0, 8)
        // The canary lives OUTSIDE `dir` so the teardown `rm -rf` cannot be what
        // makes the final assertion pass.
        const canary = `/tmp/rp-pwn-${nonce}`
        // `/` would escape the directory, the space would split the word, `;` and
        // `$( )` would start new commands, and the `touch` is a real payload with
        // an observable effect. Every one of these must survive as DATA.
        //
        // THE ORDER OF THE PAYLOAD IS DELIBERATE and was measured: the `;touch`
        // comes BEFORE the space and the `/`. With raw interpolation, the
        // journaled command's redirect target is one word, so a payload whose
        // space precedes the `;` (`rp-a b;touch …`) makes the mangled command a
        // SYNTAX ERROR — the injected `touch` never runs and the canary below
        // would be decoration that can never fire. With the `;` first, the
        // vulnerable form parses as a command LIST and the payload really
        // executes (verified against a hand-composed unquoted, unencoded command
        // on this provider: canary present). So the canary is a live detector.
        const runId = `rp-a;touch ${canary};b c/d$(x)-${nonce}`
        const threadId = `${runId}-t`
        const paths = journalPaths(runId, dir)
        const journalName = basename(dir, paths.journal)
        try {
          // This case's agent writes to STDERR as well, so the sidecar read
          // below has real bytes to compare against: an empty sidecar is also
          // what a `journalStderrReadCommand` that read the wrong path (or
          // nothing at all) would return, and that read is the only coverage
          // that command has anywhere.
          await handle.process.exec(
            journaledCommand(
              `${emitLines(['1'])}; printf 'boom\\n' 1>&2`,
              paths,
            ),
          )
          // THE SECURITY ASSERTION, and deliberately the FIRST one: nothing the
          // runId contains was executed. It is stated before the cheaper
          // structural checks below on purpose — a defect that reintroduces raw
          // interpolation would also fail the filename shape, and a case that
          // short-circuited there would never prove this probe is live rather
          // than decorative. Re-asserted after the delete, because the sweep
          // composes a DIFFERENT command (`rm -f`) from the same id.
          expect(await fileExists(handle, canary)).toBe(false)
          expect(await probeRunExit({ handle, runId, dir })).toEqual({
            state: 'finished',
            exitCode: 0,
          })
          // The encoding is what bought that: the filename carries no character
          // a shell can act on, and it stays inside the journal directory.
          expect(journalName).toMatch(/^[A-Za-z0-9._-]+\.ndjson$/)
          expect(paths.journal.startsWith(`${dir}/`)).toBe(true)

          // THE FOLLOW PATH, against this same hostile id.
          //
          // `journalFollowCommand` is the WORST command in the set under a
          // dropped `encodeRunId`: it interpolates the journal path THREE times
          // (`mkdir -p`, `: >> path`, `tail -c +N -f path`) and joins its prep
          // steps with `;` rather than `&&`, so `: >> /tmp/dir/rp-a;touch
          // <canary>;…` is a complete redirect followed by a command LIST — the
          // payload runs on EVERY attach, and a failing prep step does not stop
          // it. Nothing else reaches this command with a hostile runId: the
          // reaper's own probes are all bounded reads, and the takeover suite,
          // the only other real-provider consumer of the follow path, builds
          // alnum-only ids. So it is exercised here, where the hostile id and a
          // live canary already exist, for the cost of one read.
          //
          // The strategy is FORCED rather than capability-derived so this is the
          // follow command and not the bounded one, and the declaration is
          // checked against the live handle in both directions — a config that
          // does not describe the provider must fail rather than silently drop
          // this read.
          expect(journalReadStrategy(handle)).toBe(
            config.followUnsupported === undefined ? 'follow' : 'poll',
          )
          if (config.followUnsupported === undefined) {
            const followed: Array<string> = []
            // A backstop, so a reader that delivers nothing fails instead of
            // parking CI — a `tail -f` never ends on its own, so this read has no
            // other floor. Not the assertion — `backstopped` below proves it was
            // not what ended the loop.
            const backstop = AbortSignal.timeout(READ_BACKSTOP_MS)
            for await (const line of readJournal(handle, {
              paths,
              fromByte: 0,
              strategy: 'follow',
              signal: backstop,
            })) {
              followed.push(line.line)
              // The agent has already reached its sentinel, so this arrives; a
              // `tail -f` never ends on its own.
              if (line.line.includes(EXIT_SENTINEL_KEY)) break
            }
            // The causal witness, first: the loop must end on the sentinel
            // `break`, not on the clock. A backstopped follow read otherwise
            // reports as a one-element-vs-two array diff that says nothing about
            // why.
            expect({ backstopped: backstop.aborted }).toEqual({
              backstopped: false,
            })
            expect(followed).toEqual([
              '{"delta":"1"}',
              exitSentinelLine(paths, 0),
            ])
            expect(await fileExists(handle, canary)).toBe(false)
          }

          // The stderr SIDECAR read, which no other conformance case reaches at
          // all. Same hostile id, same canary, one `exec`.
          const sidecar = await handle.process.exec(
            journalStderrReadCommand(paths),
          )
          expect(decodeJournalRead(sidecar.stdout)).toBe('boom\n')
          expect(await fileExists(handle, canary)).toBe(false)

          // encode → journal → list → decode: the sweep's actual path back to a
          // runId, over a real `ls -1`.
          const names = await listNames(handle, dir)
          expect(names.sort()).toEqual(
            [journalName, basename(dir, paths.stderr)].sort(),
          )
          expect(decodeJournalRunId(journalName)).toEqual({
            kind: 'runId',
            runId,
          })

          const runs = new InMemoryRunStore()
          await runs.createOrResume({ runId, threadId, startedAt: Date.now() })
          await runs.update(runId, {
            status: 'completed',
            finishedAt: Date.now(),
          })
          const result = await pruneJournals({ handle, runs, dir })
          expect(result.deleted).toEqual([runId])
          expect(result.failures).toEqual([])
          expect({
            journalDeleted: !(await fileExists(handle, paths.journal)),
            sidecarDeleted: !(await fileExists(handle, paths.stderr)),
            canaryAbsent: !(await fileExists(handle, canary)),
          }).toEqual({
            journalDeleted: true,
            sidecarDeleted: true,
            canaryAbsent: true,
          })
        } finally {
          await handle.process
            .exec(`rm -f ${quote(canary)}`)
            .catch(() => undefined)
          await removeDir(handle, dir)
          await dispose()
        }
      },
    )
  })
}