@adonis-agora/agent/dist/src/rag/chunk.d.ts

Governed, durable-backed AI agent (chat + tool-calling + governance) for AdonisJS — part of the Agora ecosystem.

export interface ChunkOptions {
    /** Target max characters per chunk. Default 800. */
    chunkSize?: number;
    /** Characters of overlap carried from the end of one chunk into the next. Default 100. */
    overlap?: number;
    /**
     * The record delimiter, when the text has boundaries that *mean* something. Pass it and the chunker
     * cuts ONLY on this string — never inside a record.
     *
     * Prose has no real boundaries, so the default heuristic (paragraph → sentence → word, in the back
     * half of the window) is the right guess. Machine-shaped text is the opposite case: a spreadsheet
     * flattened to one field-labelled record per line, a log, a JSONL export. There the boundaries are
     * known exactly, and a heuristic cut lands mid-record — stranding the half that carries the row
     * identifier in a different chunk from the half that carries the value, so neither chunk can answer
     * a question about that row. `separator: '\n'` says "the lines are the records".
     *
     * Two consequences worth knowing before you pass it, both by design:
     *
     * 1. **`chunkSize` becomes a target, not a cap.** A single record longer than `chunkSize` is emitted
     *    whole, as its own over-size chunk. It is NOT split, and it does NOT fall back to the prose
     *    heuristic — a mid-record cut is the exact failure `separator` exists to prevent, so silently
     *    performing one to respect a size limit would defeat the feature. An over-size chunk is visible
     *    (you can see the record, and shorten it or raise `chunkSize`); a mangled record is not. It never
     *    drags its neighbours over the limit either: it always chunks alone.
     * 2. **`overlap` is spent on whole records.** It stops being a character count and becomes a
     *    character *budget*, filled with entire trailing records of the previous chunk — as many as fit,
     *    never a partial one, and never so many that a chunk fails to advance. Half a record is not
     *    useful context; it is the same mid-record cut by another name. `overlap: 0` carries nothing,
     *    which is usually what self-contained records want.
     *
     * Empty or whitespace-only records are dropped. An empty string is treated as "not passed" (it would
     * otherwise split every character), leaving the prose path. Omit it and NOTHING changes: the prose
     * path below is untouched, byte-for-byte, so no existing corpus is re-chunked (and so none has to be
     * re-embedded).
     */
    separator?: string;
}
/**
 * Splits text into overlapping chunks for embedding. Greedy up to `chunkSize`, but prefers to break on
 * a paragraph → sentence → word boundary in the back half of the window (so a chunk rarely cuts
 * mid-sentence), then carries `overlap` characters into the next chunk to preserve context across the
 * seam. Pure and deterministic.
 *
 * Pass {@link ChunkOptions.separator} for text whose record boundaries are known rather than guessed;
 * that path cuts only on the separator, and its two deviations (`chunkSize` becomes a target,
 * `overlap` is spent on whole records) are documented there.
 */
export declare function chunkText(text: string, options?: ChunkOptions): string[];
//# sourceMappingURL=chunk.d.ts.map