@juspay/neurolink
Version:
The pipe layer of an AI nervous system: one interface connecting provider neurons to your application, across three inference types — generate, stream and decide. `decide` returns typed, calibrated judgements from a non-generative model (~400ms, ~$0.00002
43 lines (42 loc) • 2.14 kB
TypeScript
/**
* In-turn context guard for the OpenAI-compatible chat-completions loop.
*
* `openaiChatCompletionsBase` runs its own hand-written tool loop
* (`for step < maxSteps` → `streamOneStep` → `executeToolBatch`) and appends an
* assistant tool-call message plus one `role: "tool"` message per result on
* every step. Nothing bounded that growth: the loop had no context guard of any
* kind, so a long agentic run walked into a provider "context length exceeded"
* after dozens of tool calls and lost every completed step.
*
* This is the widest-reach gap in the codebase — the chat-completions base
* backs LiteLLM, DeepSeek, OpenRouter, NVIDIA NIM, LM Studio, llama.cpp and
* every `openai-compatible` provider.
*
* The reclaim POLICY lives in `loopGuardCore` and is shared with the other
* provider loops. This module owns only the shape mapping: OpenAI-compatible
* messages in, neutral entries out, plan applied back.
*/
import type { OpenAICompatChatMessage } from "../types/index.js";
/**
* Reclaim context budget from a chat-completions loop's conversation.
*
* Returns a NEW array when it acted, or `undefined` when the loop fits and the
* history must be left byte-identical — any rewrite invalidates the provider's
* cached prompt prefix, so "no change" has to mean no change.
*
* @param observedPromptTokens the provider's reported prompt token count for
* the previous step, used to calibrate the char-based estimate for free.
* @param previousSentEstimate this guard's own estimate for the request that
* produced `observedPromptTokens`. The two must describe the SAME request.
* @param onSentEstimate receives the estimate for the request about to be sent,
* so the caller can feed it back as the next step's `previousSentEstimate`.
*/
export declare function guardOpenAICompatConversation(args: {
conversation: OpenAICompatChatMessage[];
availableInputTokens: number;
fixedOverheadTokens: number;
provider?: string;
observedPromptTokens?: number;
previousSentEstimate?: number;
onSentEstimate?: (tokens: number) => void;
}): OpenAICompatChatMessage[] | undefined;