ai/docs/03-ai-sdk-core/36-realtime.mdx

AI SDK by Vercel - build apps like ChatGPT, Claude, Gemini, and more with a single interface for any model using the Vercel AI Gateway or go direct to OpenAI, Anthropic, Google, or any other model provider.

---
title: Realtime
description: Learn how to build realtime voice conversations with the AI SDK.
---

# Realtime

<Note type="warning">Realtime is an experimental feature.</Note>

This guide covers legacy token-based, turn-based realtime conversations over
WebSockets. These sessions run in the browser and connect
directly to the provider using a short-lived token that you create on your
server. You can also route the connection through [AI Gateway](/providers/ai-sdk-providers/ai-gateway#realtime).

For OpenAI Live's continuous JSON/PCM16 WSS relay runtime and application-handled
client delegation, see
[`experimental_useRealtime`](/docs/reference/ai-sdk-ui/use-realtime#continuous-conversations).

For Mistral's managed microphone-to-speaker conversations, use
[`mistral.experimental_realtime()`](/providers/ai-sdk-providers/mistral#realtime-voice-agents)
with `api.provider`. Its provider-owned LiveKit transport handles audio, and
your server creates and closes the sessions.

The typical flow is:

1. The browser calls your setup endpoint.
1. Your server creates a short-lived realtime token with `experimental_realtime.getToken()`.
1. The browser opens a WebSocket connection to the provider or AI Gateway.
1. The model streams audio, text, and tool calls back to the browser.
1. Tool calls are handled by your application with `onToolCall`.

For continuous **OpenAI Live** conversations, use an application-owned WebSocket
relay or optional WebRTC via `api.session`. Live uses client delegation: your
application handles delegated work and submits context rather than using the
turn-based tool loop below. See the
[`experimental_useRealtime` reference](/docs/reference/ai-sdk-ui/use-realtime)
for SDP setup, server-owned permissions, capture ownership, and graceful close.

## Server-owned connections

A trusted server can own the WebSocket instead of minting a browser token. Check
`model.capabilities?.connections` for `server-websocket` and obtain the server-only
settings with `await model.getServerWebSocketConfig?.({ intent: 'conversation' })`.
OpenAI Realtime, Google, xAI and AI Gateway support this connection, alongside the
existing OpenAI Live adapter. Returned headers can contain long-lived credentials;
keep the descriptor on your server. The application owns socket lifetime,
authorization, buffering, and reconnect policy.

```ts
import { openai } from '@ai-sdk/openai';
import { WebSocket } from 'ws';

const model = openai.experimental_realtime('gpt-realtime-2.1', {
  api: 'realtime',
});
const options = { intent: 'conversation' } as const;
const { url, headers, protocols } =
  await model.getServerWebSocketConfig(options);
const socket = new WebSocket(url, protocols, { headers });
```

Feed incoming JSON to a fresh `model.createServerEventParser()` when the model
provides it, using one parser for each connection and discarding it on disconnect.
Stateless models can use `model.parseServerEvent(raw)` directly. Serialize outgoing normalized events with
`await model.serializeClientEvent(event, options)`. Use the same `intent` when
building a session configuration with `model.buildSessionConfig(config, options)`.
A complete Node example is in
[`server.ts`](https://github.com/vercel/ai/blob/main/examples/ai-functions/src/realtime/openai/server.ts).

The default intent is `conversation`. OpenAI Realtime also supports
`transcription`, which selects its dedicated endpoint and session shape. Set
`inputAudioTranscription.model` explicitly when the factory model is a conversation
model; otherwise the factory model ID is used for transcription. Instructions,
voice, tools and output settings are rejected in transcription sessions. Google,
xAI, AI Gateway and OpenAI Live reject a non-conversation intent. Google Live
translation remains a session provider option, rather than a connection intent.

### Usage snapshots

Normalized completion events can carry optional `usage` with gross `inputTokens`
and `outputTokens`, available text/audio buckets, cached input portions, and
`seconds` for duration-billed transcription. Cached counts are subsets of gross
input counts. Modality buckets are subsets of totals, so do not add them to those
totals. Unknown counts remain absent, and totals can include other modalities.
The public types are `Experimental_RealtimeServerConnection`,
`Experimental_RealtimeSessionIntent` and `Experimental_RealtimeUsage` from `ai`.

OpenAI Realtime and xAI expose reported usage on `response-done` and input
transcription completions. AI Gateway passes through normalized usage only when
the Gateway route includes it in its events.
OpenAI Live retains its existing duration snapshots on `session-usage` and
`session-closed`.

Google attaches usage to `response-done` only when the same wire frame contains
`turnComplete` and `usageMetadata`. Otherwise it emits an existing `custom` event
with `rawType: 'usageMetadata'` and typed `usage`. Google gross input totals include
reported tool-use prompt counts, and gross output totals include reported thinking
counts. These are provider snapshots;
they do not repeat completion, imply a response association, or represent increments
to sum. A later snapshot can revise an earlier one. The original provider payload
remains available in `raw`.

## Setup Endpoint

Create a setup endpoint that returns a short-lived token for the realtime
provider. This endpoint can also attach tool definitions to the session.

```ts filename='app/api/realtime/setup/route.ts'
import { openai } from '@ai-sdk/openai';
import { experimental_getRealtimeToolDefinitions, tool } from 'ai';
import { z } from 'zod';

const tools = {
  getWeather: tool({
    description: 'Get the current weather for a city',
    inputSchema: z.object({
      city: z.string().describe('The city to get weather for'),
    }),
  }),
};

export async function POST(request: Request) {
  const body = await request.json().catch(() => ({}));
  const toolDefinitions = await experimental_getRealtimeToolDefinitions({
    tools,
  });

  const token = await openai.experimental_realtime.getToken({
    model: 'gpt-realtime',
    sessionConfig: {
      ...body.sessionConfig,
      tools: toolDefinitions,
    },
  });

  return Response.json({
    ...token,
    tools: toolDefinitions,
  });
}
```

<Note>
  In production, authenticate and rate-limit your setup endpoint. It creates
  realtime sessions using your server-side API key.
</Note>

## AI Gateway

Use [AI Gateway](/providers/ai-sdk-providers/ai-gateway) when you want the same
realtime client code to work across supported upstream providers. The Gateway
normalizes realtime events server-side, and the browser still receives only a
short-lived client secret.

Create the short-lived Gateway realtime token from a server-side setup endpoint:

```ts filename='app/api/realtime/setup/route.ts'
import { gateway } from 'ai';

export async function POST() {
  const token = await gateway.experimental_realtime.getToken({
    model: 'openai/gpt-realtime-2',
  });

  return Response.json(token);
}
```

Then use the matching Gateway realtime model in the browser:

```tsx filename='app/realtime/page.tsx'
'use client';

import { experimental_useRealtime } from '@ai-sdk/react';
import { gateway } from 'ai';

const model = gateway.experimental_realtime('openai/gpt-realtime-2');
const sessionConfig = {
  instructions: 'You are a helpful assistant. Be concise.',
  inputAudioTranscription: {},
  voice: 'alloy',
  turnDetection: { type: 'server-vad' as const },
};

export default function RealtimePage() {
  const realtime = experimental_useRealtime({
    model,
    api: {
      token: '/api/realtime/setup',
    },
    sessionConfig,
  });

  // ...
}
```

<Note>
  `gateway.experimental_realtime.getToken()` must run on your server because it
  uses your Gateway credential to mint a `vcst_` client secret. Creating the
  realtime model with `gateway.experimental_realtime()` is safe in the browser.
</Note>

Tool definitions work the same way with AI Gateway: convert AI SDK tools with
`experimental_getRealtimeToolDefinitions()` in your setup endpoint and return
the definitions alongside the token. The hook includes them in the session
update after the WebSocket opens.

## Client Session

Use the `experimental_useRealtime` hook to connect to a realtime model, capture
microphone audio, play model audio, send text messages, and render messages.

```tsx filename='app/realtime/page.tsx'
'use client';

import { openai } from '@ai-sdk/openai';
import { experimental_useRealtime } from '@ai-sdk/react';

const model = openai.experimental_realtime('gpt-realtime');
const sessionConfig = {
  instructions: 'You are a helpful assistant. Be concise.',
  inputAudioTranscription: {},
  voice: 'alloy',
  turnDetection: { type: 'server-vad' as const },
};

export default function RealtimePage() {
  const realtime = experimental_useRealtime({
    model,
    api: {
      token: '/api/realtime/setup',
    },
    sessionConfig,
  });

  return (
    <div>
      <button onClick={realtime.connect}>Connect</button>
      <button onClick={realtime.disconnect}>Disconnect</button>

      {realtime.messages.map(message => (
        <div key={message.id}>
          <strong>{message.role}</strong>
          {message.parts.map((part, index) =>
            part.type === 'text' ? <span key={index}>{part.text}</span> : null,
          )}
        </div>
      ))}
    </div>
  );
}
```

Keep model and session configuration objects stable across renders. Use module
scope as above, or `useMemo` when configuration depends on props. Replacing either
object replaces the hook's session.

## Tool Calling

Realtime tool execution is client-driven. The provider sends tool calls over the
WebSocket, and your application handles them with `onToolCall`. If the result is
available immediately, return it from `onToolCall`. The SDK sends it back to the
provider as tool output.

For server-backed tools, call an app-specific API endpoint from `onToolCall`.
Avoid generic "execute tool by name" routes. App-specific endpoints are easier
to secure because they can use your normal authentication, authorization,
validation, and rate limiting rules.

### Server-Backed Tool Endpoint

```ts filename='app/api/weather/route.ts'
import { z } from 'zod';

const inputSchema = z.object({
  city: z.string(),
});

export async function POST(request: Request) {
  const input = inputSchema.safeParse(await request.json());

  if (!input.success) {
    return Response.json({ error: 'Invalid input' }, { status: 400 });
  }

  return Response.json({
    city: input.data.city,
    temperature: 72,
    condition: 'sunny',
  });
}
```

### Client Tool Handler

```tsx filename='app/realtime/page.tsx' highlight="12-26"
import { openai } from '@ai-sdk/openai';
import { experimental_useRealtime } from '@ai-sdk/react';

const model = openai.experimental_realtime('gpt-realtime');

export default function RealtimePage() {
  const realtime = experimental_useRealtime({
    model,
    api: {
      token: '/api/realtime/setup',
    },
    onToolCall: async ({ toolCall }) => {
      if (toolCall.toolName === 'getWeather') {
        const response = await fetch('/api/weather', {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify(toolCall.args),
        });

        if (!response.ok) {
          throw new Error('Weather lookup failed');
        }

        return response.json();
      }
    },
  });

  // ...
}
```

You can also submit tool output manually with `addToolOutput` when the tool
requires user interaction or another asynchronous process:

```tsx
realtime.addToolOutput(toolCallId, {
  approved: true,
});
```

## Supported Providers

The following providers support the token-based WebSocket flow above:

```ts
import { openai } from '@ai-sdk/openai';
import { google } from '@ai-sdk/google';
import { xai } from '@ai-sdk/xai';

const openaiModel = openai.experimental_realtime('gpt-realtime');
const googleModel = google.experimental_realtime(
  'gemini-3.1-flash-live-preview',
);
const xaiModel = xai.experimental_realtime('grok-voice-latest');
```

You can also route realtime through the [AI Gateway](/providers/ai-sdk-providers/ai-gateway#realtime), which normalizes the
session so the same client code works across upstream providers:

```ts
import { gateway } from '@ai-sdk/gateway';

const gatewayModel = gateway.experimental_realtime('openai/gpt-realtime-2');
```

`gateway.experimental_realtime.getToken()` mints a short-lived Gateway client
secret on your server. The browser uses that token to open the Gateway
WebSocket; the SDK handles the Gateway-specific WebSocket subprotocols for you.
See the [AI Gateway realtime docs](/providers/ai-sdk-providers/ai-gateway#realtime)
for Gateway-specific token and provider option details.

[Mistral voice agents](/providers/ai-sdk-providers/mistral#realtime-voice-agents)
use `mistral.experimental_realtime('mistral-small-latest')` with a managed session
endpoint (`api.provider`). LiveKit is handled internally by the Mistral provider.