UNPKG

@tanstack/ai-sandbox

Version:

Provider-agnostic sandbox layer for TanStack AI — run harness adapters inside isolated sandboxes (defineSandbox, defineWorkspace, withSandbox) with a uniform SandboxHandle, workspace bootstrap, policy, and resumable lifecycle.

239 lines (238 loc) 11.3 kB
import { SandboxHandle } from './contracts.js'; import { InternalLogger } from '@tanstack/ai/adapter-internals'; import { LockStore } from '@tanstack/ai/locks'; import { RunRecord, RunStatus, RunStore, StreamChunk, StreamDurability } from '@tanstack/ai'; /** * Safety net for a single run's drive. Not the mechanism that decides whether a * run finished — see the module doc for why that design was rejected — so this is * generous rather than tight: it only has to stop a drive that has genuinely * wedged on a run the journal already said was over. * * On the expiry path it is not merely a net: it is what stops a still-producing * agent, since nothing polls the cancel recorded before that drive. A caller that * expires live agents may want a tighter value there than a finalization replay * needs. */ export declare const DEFAULT_RUN_BUDGET_MS = 30000; /** * Runs one sweep will touch. A cron invocation is bounded (a Worker's CPU * budget, a Lambda timeout), and an unbounded sweep over a backlog of thousands * would be killed mid-run rather than finishing 25 and returning; the next tick * takes the next batch. */ export declare const DEFAULT_MAX_RUNS = 25; /** Journal tail bytes {@link probeRunExit} reads. The sentinel is the last line. */ export declare const DEFAULT_EXIT_PROBE_BYTES = 4096; /** * What the out-of-band probe learned about a detached run's agent. * * THREE ARMS, not a boolean, because "could not tell" must not be * indistinguishable from "still working": both leave the run alone, but only one * of them is a condition an operator should see. A two-valued probe would also * invite the caller to treat a provider `exec` failure as "finished" and drive a * live run — the exact defect this module exists to prevent. */ export type RunExitProbe = /** The `{"__exit":N}` sentinel is in the journal. The agent is over. */ { state: 'finished'; exitCode: number; } /** No sentinel. The agent is mid-flight (or never started). LEAVE IT ALONE. */ | { state: 'producing'; } /** The probe could not answer — no sandbox, `exec` rejected, frame undecodable. */ | { state: 'unknown'; error?: unknown; }; /** What one sweep did to one run. */ export type ReapRunOutcome = /** * The probe saw `{"__exit":N}`, the run was driven to a terminal status, and * its transcript is saved. The happy path. */ 'finalized' /** * Past `detachedRunTtlMs`. Cancelled first, then driven to terminal. The probe * is skipped: the outcome is terminal whether the agent finished or not. * * Reported even when {@link ReapOptions.runBudgetMs} is what ended the drive — * on this path that is the mechanism rather than an anomaly, so `'expired'` is * the truthful outcome. The run's own `status` distinguishes the two shapes: * an agent that had already finished replays to `'completed'`, while one still * producing when the budget fired is `'aborted'`. */ | 'expired' /** * Still producing. `pipeToRunLog` was NEVER entered — nothing appended, no * terminal record written, `close()` not called, `detachedSince` untouched. */ | 'producing' /** The probe could not answer. Left exactly as untouched as `'producing'`. */ | 'unknown' /** * ANOMALY. The drive outran {@link ReapOptions.runBudgetMs} on a run the * journal already said was finished. The record IS terminal and the log IS * closed (`pipeToRunLog` guarantees both), so this is a diagnostic, not a leak * — but a finished run that would not replay in 30s means the journal read, the * translation, or the log is misbehaving. * * FINALIZATION ONLY. An expired run that outran its budget reports `'expired'`: * there was no probe on that path and the agent may legitimately still have been * producing, so the budget firing is the designed stop, not a misbehaving replay. */ | 'budget-exceeded' /** * Another host holds the claim, or held it and superseded us mid-drive. Normal: * a real viewer attaching mid-sweep is exactly this. Also covers a run that * reached terminal in another host's hands between the listing and the claim. */ | 'not-claimed' /** * The transcript IS saved and the record IS terminal — only * {@link ReapOptions.reclaim} threw, so the sandbox is still up. * * A DISTINCT outcome rather than `'failed'`, because the two need opposite * operator responses and `'failed'` cannot express this one: it carries no * `status` and no `exitCode`, so "transcript saved, sandbox NOT reclaimed" * read identically to "the sweep failed and the run was never finalized". * * NOT RETRYABLE BY THE SWEEP. The record is terminal by now, so the run has * left `listReclaimable` for good; the sandbox leaks until something else * tears it down. This entry, with its `error`, is the only notice of that. * * OVERWRITES `'budget-exceeded'` when both happened, because the leak is what * needs acting on — {@link ReapRunEntry.terminalizedAnyway} is what preserves * the budget half of that pair. * * `sandboxReclaimer` REJECTS on its `'destroy-failed'` arm precisely so this * outcome is reachable through the shipped reclaimer and not only through a * custom one; see `SandboxReclaimFailedError` in `reclaim.ts`. */ | 'reclaim-failed' /** Something threw. Logged, recorded here, and the sweep continued. */ | 'failed'; /** One run's line in the sweep summary. */ export interface ReapRunEntry { runId: string; outcome: ReapRunOutcome; /** The run's status after the sweep, when the run was driven. */ status?: RunStatus; /** The agent's exit code, when the probe read one. */ exitCode?: number; /** * THE BUDGET ANOMALY MARKER, and the only field whose mere PRESENCE carries a * fact: it is set if and only if the drive outran * {@link ReapOptions.runBudgetMs} on the finalization path — the condition * `'budget-exceeded'` names. Its value is whether the record nonetheless * reached a terminal status, practically always `true` since `pipeToRunLog` is * total; it is reported rather than assumed so an operator does not have to * infer it. * * SURVIVES A FAILED RECLAIM. `reclaim` runs after the outcome is classified * and overwrites it with `'reclaim-failed'`, which is the more urgent fact (a * leaked sandbox nothing will retry) and so wins the single `outcome` slot. * This field is therefore what keeps the budget anomaly on the entry: an * operator seeing `'reclaim-failed'` WITH `terminalizedAnyway` present is * looking at a run that blew its budget and then leaked, and needs both halves. */ terminalizedAnyway?: boolean; error?: unknown; } export interface ReapResult { /** Runs in this batch — i.e. after the {@link ReapOptions.maxRuns} cap. */ considered: number; /** Runs {@link ReapOptions.hasFinished} was actually called for. */ probed: number; outcomes: Record<ReapRunOutcome, number>; runs: Array<ReapRunEntry>; } export interface ReapOptions<TOffset extends string = string> { runs: RunStore; locks: LockStore; /** * Per-run event log factory, same shape `RunDeps.durability` takes. * * Generic in the offset type, defaulted to `string` so an existing call site * needs no change — see {@link SandboxRunDriverOptions.durability} for why * hardcoding the default locked out branded-cursor backends. */ durability: (runId: string) => StreamDurability<TOffset>; /** * The out-of-band "did the agent reach its sentinel?" probe. INJECTED, because * neither the delivery log nor this package can answer it — see the module doc. * {@link probeRunExit} is the implementation an application wires in once it has * resolved the run's `SandboxHandle`. */ hasFinished: (record: RunRecord) => Promise<RunExitProbe>; /** Produce the run's remaining events. Called only once the claim is held. */ drive: (input: { runId: string; threadId: string; signal: AbortSignal; }) => AsyncIterable<StreamChunk>; /** Sweep clock, passed rather than read so a sweep is reproducible. */ now: number; /** Detached-run TTL; `detachedSince <= now - ttl` expires, INCLUSIVELY. */ detachedRunTtlMs: number; /** Safety net per drive. Defaults to {@link DEFAULT_RUN_BUDGET_MS}. */ runBudgetMs?: number; /** Batch cap. Defaults to {@link DEFAULT_MAX_RUNS}. */ maxRuns?: number; /** Quiescence window; defaults to `DEFAULT_FENCE_QUIET_MS`. */ fenceQuietMs?: number; /** * Tear the run's sandbox down. Called ONLY after the run reached a terminal * status, and with the ORIGINALLY LISTED record — see {@link reapDetachedRuns}. * `sandboxReclaimer` in `reclaim.ts` is the ready-made implementation. */ reclaim?: (record: RunRecord) => Promise<void>; logger?: InternalLogger; } /** * Read the END of a run's journal and answer whether the agent reached its * `{"__exit":N}` sentinel. Read-only: no append, no record write, no `close()`. * * This is the whole reason the reaper is safe. It is the ONLY way to learn that a * detached run is over without driving it, because the delivery log stops growing * the moment the viewer leaves while the journal does not. * * ANY failure answers `'unknown'`, never `'finished'`: the caller drives a run it * is told finished, so a provider `exec` that rejected, a sandbox that is gone, or * a frame the provider truncated must never be read as "the agent exited". * * An EMPTY tail answers `'producing'` — the fail-safe direction. A journal that * does not exist yet is indistinguishable here from one with no sentinel, and both * mean "do not touch this run". */ export declare function probeRunExit(input: { handle: SandboxHandle; runId: string; /** Journal directory; defaults to `DEFAULT_JOURNAL_DIR`, as `journalPaths` does. */ dir?: string; /** Tail bytes to read. Defaults to {@link DEFAULT_EXIT_PROBE_BYTES}. */ maxBytes?: number; }): Promise<RunExitProbe>; /** * Sweep the detached runs a `RunStore` surfaces, saving each finished run's * transcript and reclaiming its sandbox. * * A plain async function with no timer and no daemon: call it from a cron, a * queue consumer, a Durable Object `alarm()`, or a `waitUntil`. It NEVER rejects * — every failure is logged and counted in the returned {@link ReapResult}. * * ONE `listReclaimable({ now, ttlMs: 0 })` call, deliberately: `ttlMs: 0` is * every detached run, which is the candidate set for FINALIZATION (a run that hit * its sentinel one second after the viewer left has an unsaved transcript and * must not wait out the TTL), and expiry is then classified in-process against * the same inclusive cutoff. Listing twice with two TTLs would cost a second * store round-trip to compute a subset. * * `listReclaimable` is OPTIONAL on `RunStore`. A backend without it cannot be * reaped, which answers `{ considered: 0 }` plus one log line rather than * throwing — the same graceful degrade every other optional-method call site in * the repo does (`store.findActiveRun?.(threadId)`). */ export declare function reapDetachedRuns<TOffset extends string = string>(options: ReapOptions<TOffset>): Promise<ReapResult>;