agentfootprint
Version:
The explainable agent framework — backtrack a wrong answer to the exact context that caused it (evidence, not guesses). Built on footprintjs.
782 lines • 42.2 kB
JavaScript
/**
* adapters/memory/s3Vectors — the corpus outlives the runtime, and can still
* be updated without one.
*
* `sqliteVectorStore` (8.9.0) made a corpus survive a restart by putting it in
* a file, and `staticVectorStore` (8.20.0) made it survive a runtime with no
* disk at all by shipping it as a build artifact. Both leave the same gap, and
* a production field report named it: **a bundle can only change when you
* redeploy.** Add three documents on a Tuesday and the answer arrives with the
* next release train. That is a fine trade for product documentation and a bad
* one for anything a human edits during the day.
*
* Amazon S3 Vectors closes it. It is object storage with a native vector index
* and a query API: durable, serverless, nothing to run, priced like storage
* rather than like a database cluster. This adapter is the `MemoryStore` port
* over it, and the two halves that matter are:
*
* - `search()` maps 1:1 onto **QueryVectors** — the vector goes as
* `queryVector.float32`, `k` becomes `topK`, tiers become a metadata
* `filter`, and the returned `distance` becomes the port's score.
* - `put()` / `putMany()` map onto **PutVectors** — so `indexCorpus`,
* `indexFolder` and `indexDocuments` run against it unchanged. **That is
* the point of writing it rather than only reading it:** a corpus you can
* add to from a cron job at 14:00 is a different product from one you can
* add to at the next deploy.
*
* ── What it is NOT ──────────────────────────────────────────────────────────
* A vector index is not a key-value store, and this adapter refuses the
* operations that would need one rather than faking them:
*
* - `putIfVersion` — PutVectors is last-write-wins; there is no
* compare-and-set. A read-then-write here would be a compare-and-set that
* another writer can walk through the middle of, which is worse than none.
* - `recordSignature` / `feedback` — a set of strings and a running average
* are not vectors. `seen()` answers `false` and `getFeedback()` `null`,
* which are both TRUE (nothing can be recorded, so nothing has been); the
* WRITE halves refuse by name.
*
* Pair it with a second store for conversation memory — `defineRAG` for the
* corpus, `defineMemory` for the chat, each with the backend it suits. That is
* the same shape `staticVectorStore` documents.
*
* ── The index you must create first, and why this does not create it ────────
* A vector index has a DIMENSION and a DISTANCE METRIC fixed at creation, and
* both are decisions about your embedder that this library must not make on
* your behalf — creating one silently would pick a size and a metric your
* corpus then lives with. Create it once, with your infrastructure:
*
* ```bash
* aws s3vectors create-vector-bucket --vector-bucket-name my-corpus
* aws s3vectors create-index \
* --vector-bucket-name my-corpus \
* --index-name docs \
* --data-type float32 \
* --dimension 1024 \
* --distance-metric cosine \
* --metadata-configuration '{"nonFilterableMetadataKeys":["af"]}'
* ```
*
* `nonFilterableMetadataKeys: ["af"]` is load-bearing, not decoration. This
* adapter stores the whole entry — the passage the model will read, its
* provenance, its timestamps — as JSON under the metadata key `af`. Filterable
* metadata has a small per-vector budget; non-filterable metadata has the large
* one. Declare `af` non-filterable and a full passage fits; leave it filterable
* and PutVectors starts refusing your longer chunks partway through an
* indexing run.
*
* Only `ns` (the identity namespace) and `tier` are ever filtered on, and both
* are short strings.
*
* ── The preflight: this store now READS the index before trusting it (9.4.0) ─
* One `GetIndex`, lazily on first use and memoized for the life of the store.
* It checks the three things this adapter used to assume — and that a
* production deployment got wrong in all three ways at once:
*
* 1. the distance metric really is cosine (a euclidean index scored a
* self-query at 0.999…, where a true cosine is exactly 1.0);
* 2. `af` really is non-filterable, checked BEFORE the first write, so a
* misconfigured index fails at document 1 rather than at document 400
* with an AWS "Filterable metadata must have at most 2048 bytes";
* 3. the index dimension matches the vectors being written or searched.
*
* This is the discipline every sibling store already had at open —
* `sqliteVectorStore`'s schema identity, `pgVectorStore`'s schema check,
* `staticVectorStore`'s load-time fingerprint. This store had nothing to open,
* so it opened nothing.
*
* **IAM:** the caller now needs `s3vectors:GetIndex` alongside `PutVectors` /
* `QueryVectors` / `GetVectors` / `ListVectors` / `DeleteVectors`. A GetIndex
* that fails is refused by name, never passed through — an index this store
* cannot read is an index whose metric, layout and dimension it would be
* guessing at.
*
* ── Cosine only, said out loud ──────────────────────────────────────────────
* The port's score is a cosine similarity, and every threshold in this library
* — starting with `defineRAG`'s 0.7 default — is calibrated on that range. A
* cosine distance converts exactly (`score = 1 - distance`). A EUCLIDEAN
* distance does not: any mapping into [-1, 1] produces a number that READS like
* a cosine and is not one, which is the same class of confident-and-meaningless
* value the embedder-mismatch refusal exists to stop. So a euclidean index is
* refused at construction, by name.
*
* ── The fingerprint guarantee, stated exactly ───────────────────────────────
* `sqliteVectorStore` owns its file and can keep a fingerprint row in it. This
* store owns nothing but vectors, so the guarantee is assembled from what the
* service actually provides, and it is smaller — so it is spelled out rather
* than implied:
*
* - **Dimensions** are enforced by S3 Vectors itself. The index declares one;
* a vector of another length is rejected by the service, loudly, at the
* call. Nothing here can be sloppier than that.
* - **The model id** is stamped into every vector's metadata (`fp`) and
* checked in two places: against the fingerprint this process has already
* seen for the namespace (at write and at search), and against the
* fingerprint carried by the HITS that come back (at search). The second
* one is what survives a restart: the first query after an embedder swap
* sees documents stamped by the old embedder and refuses by name, instead
* of returning a confident ranking of two incompatible spaces.
* - Since 9.4.0 the index's OWN dimension is checked too, at the preflight —
* so a wrong-size embedder is caught on the first write and the first
* search of a fresh process, which the per-process fingerprint could not do
* (it has nothing to disagree with at boot).
* - What is still NOT caught: the first write of a fresh process into a
* namespace built by a different embedder OF THE SAME SIZE. Only the model
* id separates those, and it lives in the vectors rather than on the index.
* There is no cheap read that would catch it, and a full index scan on
* every boot is not one either. It is caught at the next search, before a
* single wrong answer is returned.
*
* ── Lazy peer dependency ────────────────────────────────────────────────────
* `@aws-sdk/client-s3vectors` is an OPTIONAL peer dependency, required at
* construction time. Importing `agentfootprint/memory` costs nothing for
* consumers who never build one of these. Pass `client` to share the SDK
* configuration your app already has.
*/
import { identityNamespace } from '../../memory/identity/index.js';
import { lazyRequire } from '../../lib/lazyRequire.js';
import { NO_SERVER_EXTRACTION, refuseUnhonouredWriteOptions, } from '../../memory/store/writeOptions.js';
import { EmbedderMismatchError, fingerprintConflict, fingerprintText, parseFingerprint, } from '../../lib/embedderMismatch.js';
import { storeRequestLimit } from '../../memory/store/requestLimit.js';
// ─── Metadata layout ─────────────────────────────────────────────────
/** The identity namespace, filtered on at query time. Short, so filterable. */
const NS_KEY = 'ns';
/** The tier, filtered on when the caller asks for one. Short, so filterable. */
const TIER_KEY = 'tier';
/** The embedder fingerprint, `'<id>@<dims>'`. Short, so filterable. */
const FP_KEY = 'fp';
/**
* Everything else — the passage, provenance, versions, timestamps — as one
* JSON string. Declared NON-filterable at index creation so it gets the large
* metadata budget; see the header.
*/
const PAYLOAD_KEY = 'af';
/**
* Open a `MemoryStore` over an existing S3 Vectors index.
*
* @throws when `@aws-sdk/client-s3vectors` is absent and no `client` was passed.
* @throws when `distanceMetric` is anything but `'cosine'`.
* @throws EmbedderMismatchError from `put`/`putMany`/`search` when a vector
* meets a namespace built by a different embedder.
*
* @example A corpus you can add to without a redeploy
* ```ts
* import { defineRAG, indexDocuments } from 'agentfootprint';
* import { s3VectorsStore } from 'agentfootprint/memory';
* import { bedrockEmbedder } from 'agentfootprint/providers';
*
* const store = s3VectorsStore({ bucket: 'my-corpus', index: 'docs', region: 'us-east-1' });
* const embedder = bedrockEmbedder({ region: 'us-east-1' });
*
* // Run this from a cron job. No deploy, no restart — the agent sees it next turn.
* await indexDocuments(store, embedder, newDocs, { embedderId: embedder.id });
*
* const agent = Agent.create({ provider })
* .rag(defineRAG({ id: 'docs', store, embedder, embedderId: embedder.id }))
* .build();
* ```
*/
export function s3VectorsStore(options) {
const { bucket, index } = options;
if (!bucket || bucket.trim() === '' || !index || index.trim() === '') {
throw new TypeError(`s3VectorsStore: both \`bucket\` and \`index\` are required and must be non-empty — ` +
`received bucket='${String(bucket)}', index='${String(index)}'. This store never ` +
`creates either: an index's dimension and distance metric are decisions about your ` +
`embedder, and picking them for you would be picking them forever.`);
}
const metric = options.distanceMetric ?? 'cosine';
if (metric !== 'cosine') {
throw new TypeError(`s3VectorsStore: distanceMetric '${String(metric)}' is not supported — this store ` +
`reports COSINE similarity, and every threshold in this library (starting with ` +
`defineRAG's 0.7 default) is calibrated on that range.\n` +
` A cosine distance converts exactly (score = 1 - distance). A euclidean one does ` +
`not: mapping it into [-1, 1] would produce a number that reads like a cosine and ` +
`is not one, which no threshold can separate from a real score.\n` +
` Fix: create the index with --distance-metric cosine, or rank euclidean results ` +
`in your own adapter where the units are yours to interpret.`);
}
const batchSize = Math.max(1, Math.floor(options.batchSize ?? 100));
let connection;
/**
* Resolve the client + command constructors, once, on first call.
*
* Three ways in, and the first is what keeps the tests free of AWS:
* `_client` — test injection. The SDK is never required, and each command
* is a plain object stamped with its own name, so a double
* sees exactly which call was made and with what.
* `client` — your own pre-built client; config and credentials stay yours.
* neither — this factory builds one from `region`.
*/
const connect = () => {
if (connection)
return connection;
if (options._client) {
const sdk = options._sdk;
connection = {
client: options._client,
GetIndex: sdk?.GetIndexCommand ?? shimCommand('GetIndex'),
Put: sdk?.PutVectorsCommand ?? shimCommand('PutVectors'),
Query: sdk?.QueryVectorsCommand ?? shimCommand('QueryVectors'),
Get: sdk?.GetVectorsCommand ?? shimCommand('GetVectors'),
List: sdk?.ListVectorsCommand ?? shimCommand('ListVectors'),
Delete: sdk?.DeleteVectorsCommand ?? shimCommand('DeleteVectors'),
owned: false,
};
return connection;
}
const sdk = options._sdk ?? loadS3VectorsSdk();
const missing = [
'GetIndexCommand',
'PutVectorsCommand',
'QueryVectorsCommand',
'GetVectorsCommand',
'ListVectorsCommand',
'DeleteVectorsCommand',
].filter((name) => typeof sdk[name] !== 'function');
if (missing.length > 0) {
throw new Error(`s3VectorsStore: \`@aws-sdk/client-s3vectors\` is installed but ${missing.join(', ')} ` +
`${missing.length === 1 ? 'was' : 'were'} not found. Update the SDK.`);
}
if (!options.client && typeof sdk.S3VectorsClient !== 'function') {
throw new Error('s3VectorsStore: `@aws-sdk/client-s3vectors` is installed but `S3VectorsClient` was ' +
'not found. Update the SDK, or pass `client` with a pre-built one.');
}
const owned = !options.client;
connection = {
client: options.client ??
// Checked directly above.
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
new sdk.S3VectorsClient({ ...(options.region && { region: options.region }) }),
// All six checked directly above.
/* eslint-disable @typescript-eslint/no-non-null-assertion */
GetIndex: sdk.GetIndexCommand,
Put: sdk.PutVectorsCommand,
Query: sdk.QueryVectorsCommand,
Get: sdk.GetVectorsCommand,
List: sdk.ListVectorsCommand,
Delete: sdk.DeleteVectorsCommand,
/* eslint-enable @typescript-eslint/no-non-null-assertion */
owned,
};
return connection;
};
/** Send one command. Used by the preflight itself, so it does NOT preflight. */
const rawCall = async (build, signal) => {
const conn = connect();
const command = build(conn);
return signal ? conn.client.send(command, { abortSignal: signal }) : conn.client.send(command);
};
/** Send one command, after the index has been verified once. */
const call = async (build, signal) => {
await verifyIndex(signal);
return rawCall(build, signal);
};
const scope = { vectorBucketName: bucket, indexName: index };
let verification;
const readIndex = async (signal) => {
let raw;
try {
raw = await rawCall((c) => new c.GetIndex({ ...scope }), signal);
}
catch (err) {
// Never a silent pass-through: an index this store cannot READ is an
// index whose metric, layout and dimension it is guessing at, and it
// guessed wrong three times in the field.
throw new Error(`s3VectorsStore: could not read the index '${bucket}/${index}' to verify it.\n` +
` ${err instanceof Error ? err.message : String(err)}\n` +
` This store reads the index ONCE before its first call, to check the distance ` +
`metric, the metadata layout and the dimension — three things it used to assume, ` +
`and that a production deployment got wrong in all three ways at once.\n` +
` Check: the index exists in this region; the caller has the \`s3vectors:GetIndex\` ` +
`permission (PutVectors/QueryVectors alone are not enough since 9.4.0); the bucket ` +
`and index names are right.`);
}
const info = raw?.index ?? {};
const metadata = info['metadataConfiguration'];
const keys = Array.isArray(metadata?.nonFilterableMetadataKeys)
? (metadata?.nonFilterableMetadataKeys).map(String)
: [];
return {
...(typeof info['distanceMetric'] === 'string' && {
distanceMetric: info['distanceMetric'],
}),
...(typeof info['dimension'] === 'number' && { dimension: info['dimension'] }),
nonFilterableMetadataKeys: keys,
};
};
/**
* Read the index once and refuse anything this store cannot serve honestly.
*
* Memoized on the PROMISE, so concurrent first calls share one GetIndex and
* one verdict. A refusal is memoized too — re-asking a broken index on every
* operation would turn one clear failure into a storm of them.
*/
const verifyIndex = async (signal) => {
verification ??= readIndex(signal).then((found) => {
// (1) The metric. `options.distanceMetric` was a declaration; this is the
// index. A declaration that disagrees with the index is worse than none,
// so the message says which said what.
if (found.distanceMetric !== undefined && found.distanceMetric.toLowerCase() !== 'cosine') {
throw new TypeError(`s3VectorsStore: the index '${bucket}/${index}' was created with distance metric ` +
`'${found.distanceMetric}', and this store reports COSINE similarity.\n` +
(options.distanceMetric !== undefined
? ` You declared '${options.distanceMetric}'. The index disagrees, and the index ` +
`is the one that ranks.\n`
: '') +
` A cosine distance converts exactly (score = 1 - distance). A euclidean one does ` +
`not: mapped into [-1, 1] it produces a number that READS like a cosine and is ` +
`not one — a self-query scores 0.999… where a real cosine is exactly 1.0, which ` +
`no threshold (starting with defineRAG's 0.7 default) can separate from a real ` +
`score.\n` +
` Fix: recreate the index with --distance-metric cosine, or rank euclidean ` +
`results in your own adapter where the units are yours to interpret.`);
}
return found;
});
return verification;
};
/**
* (2) The metadata layout, checked before the first byte is written.
*
* Separate from the metric because the failure it prevents is a WRITE
* failure: with `af` filterable, short entries go in and longer ones are
* refused by the service partway through an import — a corpus that is 60%
* indexed and reports success for the part that fit.
*/
const assertWritableLayout = (found) => {
if (found.nonFilterableMetadataKeys.includes(PAYLOAD_KEY))
return;
throw new TypeError(`s3VectorsStore: the index '${bucket}/${index}' does not declare '${PAYLOAD_KEY}' as ` +
`non-filterable metadata, so this store cannot write to it.\n` +
` It stores each entry — the passage the model will read, its provenance, its ` +
`timestamps — as JSON under the metadata key '${PAYLOAD_KEY}'. Filterable metadata ` +
`has a small per-vector budget (2048 bytes); non-filterable has the large one. ` +
`Without this, an import writes the short chunks, then fails on a longer one with ` +
`"Filterable metadata must have at most 2048 bytes" — partway through, having ` +
`already reported success for everything before it.\n` +
` The index declares: ${found.nonFilterableMetadataKeys.length === 0
? '(no non-filterable keys)'
: found.nonFilterableMetadataKeys.join(', ')}\n` +
` Fix: recreate the index with\n` +
` --metadata-configuration '{"nonFilterableMetadataKeys":["${PAYLOAD_KEY}"]}'\n` +
` This cannot be changed after creation, so the index has to be rebuilt.`);
};
/**
* (3) The dimension, against the embedder that is actually being used.
*
* Same rule `sqliteVectorStore` applies and for the same reason: DIMENSIONS
* ALWAYS DECIDE. The service would reject the vector anyway — one write at a
* time, saying nothing about which embedder is wrong — where this names both
* numbers and the two ways out.
*/
const assertDimension = (found, dims, operation) => {
if (found.dimension === undefined || found.dimension === dims)
return;
throw new EmbedderMismatchError(`${bucket}/${index}`, `the index@${found.dimension}`, `this embedder@${dims}`, 'dimensions', operation, 'point this store at an index whose dimension matches this embedder');
};
/** Fingerprints this process has seen, one per namespace. */
const fingerprints = new Map();
let closed = false;
const open = (verb) => {
if (closed) {
throw new Error(`[memory] the s3VectorsStore for '${bucket}/${index}' is closed, so it cannot ` +
`${verb}. close() is final by design — reopening the client behind you would hide ` +
`a shutdown-ordering bug rather than surface it. Build a new store if you need one ` +
`after closing this.`);
}
};
/**
* Compare an arriving fingerprint against the one this process holds, refuse
* a real conflict, and adopt the arriving one when it is the first or the
* more specific of the two. The same rule `sqliteVectorStore` applies — over
* a smaller record, which the header states precisely.
*/
const reconcile = (ns, incoming, operation) => {
const storedText = fingerprints.get(ns);
if (storedText === undefined) {
fingerprints.set(ns, fingerprintText(incoming));
return;
}
const conflict = fingerprintConflict(parseFingerprint(storedText), incoming);
if (conflict !== null) {
throw new EmbedderMismatchError(ns, storedText, fingerprintText(incoming), conflict, operation, 'point this store at a different index');
}
if (parseFingerprint(storedText).id === undefined && incoming.id !== undefined) {
fingerprints.set(ns, fingerprintText(incoming));
}
};
const store = {
bucket,
index,
// Vectors in, ranked vectors out — QueryVectors ranks the embeddings this
// store was handed, so the corpus builders may write into it.
supportsVectorSearch: true,
ranksBy: 'vector',
fingerprintOf(identity) {
return fingerprints.get(identityNamespace(identity));
},
async get(identity, id) {
open('read an entry');
const ns = identityNamespace(identity);
const out = (await call((c) => new c.Get({ ...scope, keys: [vectorKey(ns, id)] })));
const row = out?.vectors?.[0];
if (!row)
return null;
const entry = vectorToEntry(row);
// Expired reads as absent, the same as every other store. No access
// counters: S3 Vectors has no in-place metadata update, so bumping one
// would mean rewriting the vector on every read.
if (!entry || isExpired(entry.ttl))
return null;
return entry;
},
// Write options: none honoured (refused by name in putMany, below).
async put(identity, entry, options) {
await this.putMany(identity, [entry], options);
},
async putMany(identity, entries, options) {
open('write entries');
// The port requires an empty batch to be a no-op — callers rely on it to
// skip a round-trip on a turn that produced nothing.
if (entries.length === 0)
return;
// Nothing here is ever extracted, so a write option is refused by name —
// never dropped, which would read exactly like honouring it.
refuseUnhonouredWriteOptions('s3VectorsStore', undefined, options, NO_SERVER_EXTRACTION);
const ns = identityNamespace(identity);
// Before a byte: the index really is cosine, and it really has room for a
// passage. Memoized, so this is one GetIndex for the life of the store —
// and it happens HERE rather than inside `call` below, because a layout
// that cannot hold an entry must stop the import at document 1, not at
// document 400 with 399 already written.
const found = await verifyIndex();
assertWritableLayout(found);
const vectors = entries.map((entry) => {
const embedding = entry.embedding;
if (!embedding || embedding.length === 0) {
throw new Error(`s3VectorsStore.putMany: entry '${entry.id}' carries no \`embedding\`, and a vector ` +
`index has nowhere to put an entry without one.\n` +
` Every write here must arrive already embedded — which is what indexDocuments, ` +
`indexCorpus and indexFolder do. A conversation-memory pipeline over an ` +
`un-embedded turn belongs in a store built for key-value writes (RedisStore, ` +
`AgentCoreStore), registered separately with defineMemory.`);
}
const fp = {
...(entry.embeddingModel !== undefined && { id: entry.embeddingModel }),
dims: embedding.length,
};
assertDimension(found, embedding.length, 'write to');
reconcile(ns, fp, 'write to');
return {
key: vectorKey(ns, entry.id),
data: { float32: [...embedding] },
metadata: {
[NS_KEY]: ns,
...(entry.tier !== undefined && { [TIER_KEY]: entry.tier }),
[FP_KEY]: fingerprintText(fp),
[PAYLOAD_KEY]: JSON.stringify(payloadOf(entry)),
},
};
});
// Chunked well under the service limit. PutVectors is not transactional
// across chunks and this does not pretend otherwise: a failure part-way
// throws with the chunks before it already written, which is exactly what
// the port says about batches. The corpus builders are id-deterministic,
// so re-running the same index call converges.
for (let i = 0; i < vectors.length; i += batchSize) {
const chunk = vectors.slice(i, i + batchSize);
await call((c) => new c.Put({ ...scope, vectors: chunk }));
}
},
async putIfVersion() {
return refuse('putIfVersion', 'PutVectors is last-write-wins; S3 Vectors has no compare-and-set', 'a read-then-write here would be a compare-and-set another writer can walk through ' +
'the middle of, which is worse than not offering one', 'use put()/putMany() when you are the only writer, or keep the version-critical ' +
'entries in a store with real conditional writes (RedisStore) and the corpus here');
},
async list(identity, listOptions) {
open('list entries');
const limit = storeRequestLimit(listOptions?.limit, 'limit') ?? 100;
const ns = identityNamespace(identity);
const prefix = `${ns}#`;
const tierFilter = listOptions?.tiers ? new Set(listOptions.tiers) : undefined;
const page = [];
// ListVectors has no key-prefix filter, so a namespace is selected HERE,
// over pages of the whole index. That is honest but not free: on an index
// holding many namespaces this reads more than it returns. One index per
// corpus is the shape this store is for, and `search()` — the hot path —
// filters server-side on `ns` and never does this.
let token = listOptions?.cursor;
do {
const out = (await call((c) => new c.List({
...scope,
maxResults: Math.min(500, Math.max(limit, 100)),
returnMetadata: true,
returnData: true,
...(token !== undefined && { nextToken: token }),
})));
token = out?.nextToken;
for (const row of out?.vectors ?? []) {
if (typeof row.key !== 'string' || !row.key.startsWith(prefix))
continue;
const entry = vectorToEntry(row);
if (!entry || isExpired(entry.ttl))
continue;
if (tierFilter && (entry.tier === undefined || !tierFilter.has(entry.tier)))
continue;
page.push(entry);
if (page.length === limit)
break;
}
} while (page.length < limit && token !== undefined);
// The cursor is the service's own continuation token, so a caller pages
// through the INDEX rather than through our filtered view — which is why
// it is only returned when the service says there is more.
return { entries: page, ...(token !== undefined && { cursor: token }) };
},
async delete(identity, id) {
open('delete an entry');
const ns = identityNamespace(identity);
await call((c) => new c.Delete({ ...scope, keys: [vectorKey(ns, id)] }));
},
async seen() {
// TRUE, not a stub: nothing can be recorded here (see `recordSignature`),
// so nothing has been.
return false;
},
async recordSignature() {
return refuse('recordSignature', 'a recognition set is a set of strings, and this index stores vectors', 'accepting the signature and dropping it would make seen() answer false forever ' +
'while looking like a working dedup', 'give the write pipeline a store with a set primitive (RedisStore) and keep the ' +
'corpus here — a corpus registered through defineRAG is read-only and never ' +
'records signatures at all');
},
async feedback() {
return refuse('feedback', 'a running usefulness average needs a read-modify-write this index cannot do', 'silently dropping the signal would leave getFeedback() answering null forever ' +
'while the calls that fed it looked successful', 'record usefulness in a store built for counters (RedisStore), or collect it in ' +
'your own telemetry from the agentfootprint.memory.retrieved events');
},
async getFeedback() {
// TRUE, not a stub: no feedback can be recorded here, so none has been.
return null;
},
async forget(identity) {
open('forget a namespace');
const ns = identityNamespace(identity);
const prefix = `${ns}#`;
let token;
do {
const out = (await call((c) => new c.List({
...scope,
maxResults: 500,
...(token !== undefined && { nextToken: token }),
})));
token = out?.nextToken;
const keys = (out?.vectors ?? [])
.map((v) => v.key)
.filter((k) => typeof k === 'string' && k.startsWith(prefix));
// Deleted page by page rather than collected and deleted at the end: a
// GDPR erasure that fails half way should have erased half, not
// nothing.
for (let i = 0; i < keys.length; i += batchSize) {
await call((c) => new c.Delete({ ...scope, keys: keys.slice(i, i + batchSize) }));
}
} while (token !== undefined);
fingerprints.delete(ns);
},
async search(identity, query, searchOptions) {
open('search');
const k = storeRequestLimit(searchOptions?.k, 'k') ?? 10;
const ns = identityNamespace(identity);
if (query.length === 0)
return [];
// The index's own dimension, which outranks anything this process has
// merely SEEN: a fresh process searching an index built for another
// embedder has no stored fingerprint to disagree with.
assertDimension(await verifyIndex(), query.length, 'search');
// Refused BEFORE the call, against what this process has seen — and
// again below, against what actually comes back.
reconcile(ns, {
...(searchOptions?.embedderId !== undefined && { id: searchOptions.embedderId }),
dims: query.length,
}, 'search');
const tiers = searchOptions?.tiers;
const nsClause = { [NS_KEY]: { $eq: ns } };
const filter = tiers && tiers.length > 0
? { $and: [nsClause, { [TIER_KEY]: { $in: [...tiers] } }] }
: nsClause;
const out = (await call((c) => new c.Query({
...scope,
queryVector: { float32: [...query] },
topK: k,
filter,
returnMetadata: true,
returnDistance: true,
})));
const incoming = {
...(searchOptions?.embedderId !== undefined && { id: searchOptions.embedderId }),
dims: query.length,
};
const now = Date.now();
const minScore = searchOptions?.minScore;
const scored = [];
for (const row of out?.vectors ?? []) {
// The restart-surviving half of the fingerprint check: these documents
// carry the fingerprint they were written with, and a swap shows up
// here, on the first query, before a single ranking is returned.
const storedFp = readMetaString(row, FP_KEY);
if (storedFp !== undefined) {
const conflict = fingerprintConflict(parseFingerprint(storedFp), incoming);
if (conflict !== null) {
throw new EmbedderMismatchError(ns, storedFp, fingerprintText(incoming), conflict, 'search', 'point this store at a different index');
}
}
const entry = vectorToEntry(row);
if (!entry)
continue;
if (entry.ttl !== undefined && entry.ttl <= now)
continue;
// Cosine distance to cosine similarity. Exact, not a rescaling — which
// is why a euclidean index is refused at construction rather than
// approximated here.
const distance = typeof row.distance === 'number' ? row.distance : undefined;
if (distance === undefined)
continue;
const score = 1 - distance;
if (minScore !== undefined && score < minScore)
continue;
scored.push({ entry, score });
}
// S3 Vectors returns its own ranking; re-sort anyway so the port's
// "descending by score" holds even if a future API version reorders.
scored.sort((a, b) => {
if (b.score !== a.score)
return b.score - a.score;
return a.entry.id < b.entry.id ? -1 : a.entry.id > b.entry.id ? 1 : 0;
});
return scored.slice(0, k);
},
close() {
if (closed)
return;
closed = true;
if (connection?.owned)
connection.client.destroy?.();
},
};
return store;
}
// ─── Refusals ────────────────────────────────────────────────────────
/** One shape for every "a vector index cannot do this" refusal. */
function refuse(method, because, why, fix) {
throw new Error(`s3VectorsStore.${method}: not supported — ${because}.\n` +
` This refuses rather than accepting the call: ${why}.\n` +
` Fix: ${fix}.`);
}
// ─── Keys, payloads, rows ────────────────────────────────────────────
/**
* `'<namespace>#<id>'`.
*
* The namespace is in the KEY as well as in the metadata, and the redundancy is
* deliberate: `search` filters server-side on the metadata, while `list` and
* `forget` need to select a namespace out of a page of keys, which ListVectors
* gives no filter for.
*/
function vectorKey(namespace, id) {
return `${namespace}#${id}`;
}
/** The entry id back out of a key. Ids may contain `#` (`'refunds.md#3'`); namespaces may not. */
function idFromKey(key, prefixLength) {
return key.slice(prefixLength);
}
function payloadOf(entry) {
// The embedding is deliberately NOT in here: it is the vector itself, and
// storing it twice would double the metadata budget for nothing.
return {
id: entry.id,
value: entry.value ?? null,
...(entry.metadata !== undefined && { metadata: { ...entry.metadata } }),
version: entry.version,
createdAt: entry.createdAt,
updatedAt: entry.updatedAt,
lastAccessedAt: entry.lastAccessedAt,
accessCount: entry.accessCount,
...(entry.ttl !== undefined && { ttl: entry.ttl }),
...(entry.tier !== undefined && { tier: entry.tier }),
...(entry.source !== undefined && { source: entry.source }),
...(entry.embeddingModel !== undefined && { embeddingModel: entry.embeddingModel }),
};
}
function readMetaString(row, key) {
const value = row.metadata?.[key];
return typeof value === 'string' ? value : undefined;
}
/**
* A returned vector back into a `MemoryEntry`, or `undefined` when the row
* carries no payload this store wrote.
*
* A row without `af` is not corruption to shout about — it is somebody else's
* vector in a shared index — so it is skipped rather than thrown on. A row
* whose `af` will not parse is the same fact one level down.
*/
function vectorToEntry(row) {
const raw = readMetaString(row, PAYLOAD_KEY);
if (raw === undefined)
return undefined;
let payload;
try {
payload = JSON.parse(raw);
}
catch {
return undefined;
}
const ns = readMetaString(row, NS_KEY);
const id = typeof payload.id === 'string' && payload.id.length > 0
? payload.id
: typeof row.key === 'string' && ns !== undefined
? idFromKey(row.key, ns.length + 1)
: undefined;
if (id === undefined)
return undefined;
const vector = Array.isArray(row.data) ? row.data : row.data?.float32;
return {
id,
value: payload.value,
...(payload.metadata !== undefined && { metadata: payload.metadata }),
version: payload.version ?? 1,
createdAt: payload.createdAt ?? 0,
updatedAt: payload.updatedAt ?? 0,
lastAccessedAt: payload.lastAccessedAt ?? 0,
accessCount: payload.accessCount ?? 0,
...(payload.ttl !== undefined && { ttl: payload.ttl }),
...(payload.tier !== undefined && { tier: payload.tier }),
...(payload.source !== undefined && { source: payload.source }),
...(payload.embeddingModel !== undefined && { embeddingModel: payload.embeddingModel }),
...(Array.isArray(vector) && vector.length > 0 && { embedding: [...vector] }),
};
}
function isExpired(ttl) {
return ttl !== undefined && ttl <= Date.now();
}
// ─── SDK loading ─────────────────────────────────────────────────────
/** A no-op command shim: an injected `_client` receives the input, stamped with its name. */
function shimCommand(name) {
return class {
__command = name;
constructor(input) {
Object.assign(this, input);
}
};
}
function loadS3VectorsSdk() {
try {
return lazyRequire('@aws-sdk/client-s3vectors');
}
catch {
throw new Error('s3VectorsStore requires the `@aws-sdk/client-s3vectors` peer dependency.\n' +
' Install: npm install @aws-sdk/client-s3vectors\n' +
' Or pass `client` with a pre-built S3 Vectors client.\n' +
' There is deliberately no fallback to an in-memory index: a corpus that forgot ' +
'every document on restart looks, from the outside, exactly like one that was ' +
'never built.');
}
}
//# sourceMappingURL=s3Vectors.js.map