@juspay/neurolink/dist/adapters/videoFormatSupport.d.ts

The pipe layer of an AI nervous system: one interface connecting provider neurons to your application, across three inference types — generate, stream and decide. `decide` returns typed, calibrated judgements from a non-generative model (~400ms, ~$0.00002

/**
 * Native video delivery to providers that can watch.
 *
 * ## The gap this closes
 *
 * Attaching a video produced a metadata block and a handful of downsampled
 * JPEGs:
 *
 *   ## Video File: "demo.mp4"
 *   Duration: 0:47 | 1920x1080 | avc | 30 fps | 3 keyframes extracted
 *
 * Every provider got the same treatment, Gemini included — and Gemini has
 * accepted inline video, with its audio track, the whole time. A 47-second
 * screen recording reached it as three still frames at 768px, so anything
 * between them (a click, a transition, a spoken sentence) was simply not in
 * the request.
 *
 * Two things make that worse than the equivalent audio gap was. First, the
 * frames come from ffmpeg, and ffmpeg is an optional dependency: on a machine
 * without it `extractKeyframes` returns an empty array and the model receives
 * the metadata line alone — a video attachment that conveys nothing visual at
 * all, with no error. Second, the metadata block answers exactly the questions
 * a test tends to ask ("how long is it?", "what resolution?"), so the failure
 * reads as working support until someone asks what happens in the clip.
 *
 * Delivering the file itself needs no ffmpeg, which is what makes this the
 * fix rather than "extract more frames".
 *
 * ## Provider scope
 *
 * A capability table rather than "send video to everyone". A provider that
 * cannot accept a video part answers with an opaque HTTP 400, which is
 * strictly worse than the metadata-plus-frames summary it would otherwise
 * have received — so an unlisted provider keeps the existing behaviour and
 * loses nothing.
 *
 * ## Why there is no transcode path here
 *
 * `audioFormatSupport` re-encodes a container Gemini will not read, because a
 * voice memo is seconds of audio and the conversion is cheap. The video
 * equivalent is not: re-encoding a half-hour recording is minutes of CPU
 * inside a generation request, for a payload that will usually breach the
 * inline ceiling anyway. An unsupported container therefore falls back to
 * keyframes, which is the behaviour that already existed.
 *
 * @module adapters/videoFormatSupport
 */
import type { MultimodalVideoEntry, VideoDeliveryDecision, VideoProviderConfig } from "../types/index.js";
/**
 * Per-provider video handling.
 *
 * Keys are lowercase canonical names and the aliases each provider is
 * addressed by elsewhere in the codebase; `getVideoProviderConfig` normalises
 * before looking up, so a caller never has to know which spelling it holds.
 *
 * Only Google's Gemini front ends carry `supportsNativeVideo: true`. The rest
 * are listed deliberately rather than left to the unknown-provider default:
 * an explicit row is how `getVideoProviderConfig` distinguishes "this provider
 * takes frames" from "nobody has looked at this provider yet", and the frame
 * budgets differ enough between them to be worth stating.
 */
export declare const VIDEO_PROVIDER_CONFIGS: Readonly<Record<string, VideoProviderConfig>>;
/**
 * The video-handling row for `provider`, or null when there is none.
 *
 * Null means "not described here", which is not the same as "takes frames":
 * an unrecognised provider still receives keyframes, because that is the
 * pipeline's default, but nothing in this table asserts it will understand
 * them. Callers wanting the safe reading should treat null as no native
 * video, which is what {@link supportsNativeVideo} does.
 *
 * `model` disambiguates Vertex: `GoogleVertexProvider` sends any model whose
 * id contains "claude" to the native Anthropic SDK, which never sees the
 * video part at all, so a Claude-on-Vertex request gets the frame-extraction
 * row (the same one plain `"anthropic"` gets) rather than the Gemini row the
 * provider name alone would suggest. Omitting `model` keeps the previous,
 * provider-only lookup, which is still correct for every non-Vertex caller —
 * AI Studio and the bare `"gemini"` alias never host Claude models.
 */
export declare function getVideoProviderConfig(provider: string, model?: string): VideoProviderConfig | null;
/** Whether `provider` (optionally, running `model`) can be handed raw video
 * bytes. See {@link getVideoProviderConfig} for the Claude-on-Vertex case. */
export declare function supportsNativeVideo(provider: string, model?: string): boolean;
/** Whether `mimeType` is a container a native provider will read as-is. */
export declare function isNativeVideoMimeType(mimeType: string): boolean;
/**
 * Rough token cost of putting one video in front of one provider.
 *
 * Two quite different prices, because two quite different payloads: a native
 * provider is billed for the clip's duration, a frame-extraction provider for
 * the frames it is sent. Asking for one number without saying which mode
 * applies is how a budget ends up an order of magnitude out, so the provider
 * decides the formula rather than the caller.
 *
 * An estimate, not a quote — resolution settings, prompt text and provider
 * pricing changes all move the real figure.
 *
 * @param options.provider - Provider the video is destined for.
 * @param options.durationSec - Clip length. 0 when unknown.
 * @param options.frameCount - Frames that would be extracted. Defaults to the
 *   provider's `recommendedFrameCount`, or 8 for an unlisted provider.
 * @param options.hasTranscription - Whether a speech transcript is included.
 *   Ignored for a native provider, which already hears the audio track.
 */
export declare function estimateVideoTokens(options: {
    provider: string;
    durationSec: number;
    frameCount?: number;
    hasTranscription?: boolean;
}): number;
/**
 * Whether one specific clip may go to one specific provider as bytes.
 *
 * Every rejection carries a reason the caller can log verbatim, because the
 * user-visible symptom of all of them is identical — "it only described the
 * file" — and the remedies are not: shorten the clip, re-encode the
 * container, or switch provider.
 *
 * An unknown duration is not a rejection. Probing fails on exotic containers
 * and on machines without ffmpeg, and refusing a 2 MB clip because nothing
 * measured it would reintroduce the frames-only behaviour precisely where
 * frames are least likely to be available.
 *
 * @param priorNativeVideoBytes - Source bytes already committed to the
 *   *same request's* inline budget: every inline image (keyframes
 *   included), PDF and audio part assembled before the clips, plus each
 *   clip accepted earlier. The caller accumulates this running total, since
 *   this function is otherwise stateless and cannot see the rest of the
 *   request. Defaults to 0, which makes the size check below a pure
 *   per-clip check.
 * @param model - See {@link getVideoProviderConfig}: disambiguates a
 *   Claude-on-Vertex request from a real Gemini-on-Vertex one.
 */
export declare function canDeliverVideoNatively(provider: string, video: MultimodalVideoEntry, priorNativeVideoBytes?: number, model?: string): VideoDeliveryDecision;