ai
Version:
AI SDK by Vercel - build apps like ChatGPT, Claude, Gemini, and more with a single interface for any model using the Vercel AI Gateway or go direct to OpenAI, Anthropic, Google, or any other model provider.
433 lines (350 loc) • 18.9 kB
text/mdx
---
title: Transcription
description: Learn how to transcribe audio with the AI SDK.
keywords:
- speech to text
- audio transcription
---
# Transcription
The AI SDK provides the [`transcribe`](/docs/reference/ai-sdk-core/transcribe)
function to transcribe audio using a transcription model.
```ts
import { transcribe } from 'ai';
import { openai } from '@ai-sdk/openai';
import { readFile } from 'fs/promises';
const transcript = await transcribe({
model: openai.transcription('whisper-1'),
audio: await readFile('audio.mp3'),
});
```
The `audio` property can be a `Uint8Array`, `ArrayBuffer`, `Buffer`, `string` (base64 encoded audio data), or a `URL`.
To access the generated transcript:
```ts
const text = transcript.text; // transcript text e.g. "Hello, world!"
const segments = transcript.segments; // array of segments with start and end times, if available
const words = transcript.words; // word timings, when the provider returns them separately from segments
const language = transcript.language; // language of the transcript e.g. "en", if available
const durationInSeconds = transcript.durationInSeconds; // duration of the transcript in seconds, if available
```
## Word and segment timings
Request word timings for highlighting and segment timings for captions with
`timestampGranularities`:
```ts
const transcript = await transcribe({
model: openai.transcription('whisper-1'),
audio: await readFile('audio.mp3'),
timestampGranularities: ['word', 'segment'],
});
console.log(transcript.words);
console.log(transcript.segments);
console.log(transcript.warnings);
```
The OpenAI, Azure Speech, Vertex Gemini, and xAI adapters support this option.
Explicit provider settings take precedence over the shared request. These
adapters warn when a requested granularity is unavailable or disabled by a
provider setting. Other adapters can be updated independently; use their
provider options for timing controls until they support the shared option.
`words` is optional and contains timings only when exposed by the adapter.
Timings can also be returned by default, without an explicit request.
`segments` keeps its existing provider-specific behavior: Vertex Gemini and
xAI return words there, and OpenAI falls back to words when no segments are
returned. The SDK does not construct caption segments from word timings.
Omitting `timestampGranularities` keeps provider defaults. An empty array
requests no additional timing detail; it does not suppress timings returned by
default. Older V2 and V3 model adapters warn that they do not support the shared
option.
AI Gateway currently warns that it does not forward this shared timing option.
Use provider-specific timing controls in `providerOptions`, or use a direct
provider model instance. Gateway exposes `words` when its response includes
normalized word timings.
## Telemetry
`transcribe` and `experimental_streamTranscribe` accept the standard
`telemetry` option. Registered integrations receive start, end, and error
events with the provider and model, audio byte length and media type, final
transcript, warnings, response metadata, provider metadata, and usage when the
provider reports it. Because streaming transcription is experimental, custom
integrations receive its lifecycle through
`experimental_onStreamTranscriptionStart` and
`experimental_onStreamTranscriptionEnd`; errors continue to use `onError`.
Integrations that wrap or delegate to another telemetry integration must
forward these experimental callbacks explicitly for streaming transcription
to be observed.
```ts highlight="5-7"
const transcript = await transcribe({
model: openai.transcription('gpt-4o-transcribe'),
audio: await readFile('audio.mp3'),
telemetry: {
functionId: 'transcribe-customer-call',
},
});
```
Raw audio is not included in telemetry events. For streaming transcription,
the final byte length is emitted only after the input stream is consumed and
the transcription finishes. Use `recordInputs: false` or
`recordOutputs: false` to prevent integrations from recording sensitive input
or output fields.
For URL audio downloaded by the SDK, the downloaded media type is passed to the provider and reported
on successful telemetry events. When a provider fetches a URL natively, audio byte length and media type are undefined because the SDK does not inspect the audio. If downloading or decoding input audio fails,
the start event is still emitted before the correlated error event, but its
audio byte length and media type can be undefined because preparation did not
complete.
## Streaming Transcription
<Note type="warning">Streaming transcription is an experimental feature.</Note>
Use `experimental_streamTranscribe` when you have live raw audio and need transcript updates before the full audio stream is complete. The function uses transcription models with streaming support; provider options configure provider-specific behavior, but the streaming operation is selected by the function itself.
```ts
import { openai } from '@ai-sdk/openai';
import { experimental_streamTranscribe as streamTranscribe } from 'ai';
const result = streamTranscribe({
model: openai.transcription('gpt-realtime-whisper'),
audio: audioStream, // ReadableStream<Uint8Array | string>
inputAudioFormat: { type: 'audio/pcm', rate: 24000 },
providerOptions: {
openai: {
language: 'en',
streaming: {
delay: 'low',
},
},
},
});
for await (const part of result.fullStream) {
if (part.type === 'transcript-delta') {
process.stdout.write(part.delta);
}
if (part.type === 'transcript-partial') {
console.log('partial:', part.text);
}
if (part.type === 'transcript-final') {
console.log('final:', part.text);
}
}
console.log(await result.text);
```
`fullStream` is a single-consumer live stream and can only be accessed once.
When you need both stream parts and final results, access `fullStream` first and
await the result promises while or after consuming it. Accessing a result
promise first consumes the stream internally, so `fullStream` is no longer
available. This avoids retaining an unbounded replay buffer for live audio.
To access the final transcript metadata:
```ts
const text = await result.text; // final transcript text
const segments = await result.segments; // final segments with timing, if available
const language = await result.language; // language of the transcript, if available
const durationInSeconds = await result.durationInSeconds; // duration in seconds, if available
```
The `audio` stream must contain raw audio chunks. `Uint8Array` chunks are raw bytes; `string` chunks are base64-encoded raw bytes. Always set `inputAudioFormat` to match the chunks you send.
<Note>
String model IDs resolve through the global provider (AI Gateway by
default). AI Gateway supports streaming transcription for supported models
(e.g. `openai/gpt-realtime-whisper`, `elevenlabs/eleven-scribe-2-realtime`,
`xai/grok-stt`), so string IDs work:
`experimental_streamTranscribe({ model: 'openai/gpt-realtime-whisper', ...
})`. You can also pass a provider model instance (e.g.
`openai.transcription('gpt-realtime-whisper')`) to stream directly against
the provider.
</Note>
Deepgram Nova models support streaming through
`deepgram.transcription('nova-3')`; see the
[Deepgram streaming documentation](/providers/ai-sdk-providers/deepgram#streaming-transcription)
for audio formats and provider options.
OpenAI streaming transcription uses `openai.transcription('gpt-realtime-whisper')`.
Cartesia uses `cartesia.transcription('ink-2')` for streaming-only Ink 2
transcription. ElevenLabs uses
`elevenLabs.transcription('scribe_v2_realtime')` for Scribe v2 Realtime. Azure
uses `azure.transcription('mai-transcribe-2-streaming')` for streaming-only
MAI-Transcribe-2-Streaming. xAI
uses the same `xai.transcription()` model for request/response and streaming
transcription; `experimental_streamTranscribe` selects the provider's
WebSocket STT transport.
```ts
import { xai } from '@ai-sdk/xai';
import { experimental_streamTranscribe as streamTranscribe } from 'ai';
const result = streamTranscribe({
model: xai.transcription(),
audio: audioStream,
inputAudioFormat: { type: 'audio/pcm', rate: 16000 },
providerOptions: {
xai: {
language: 'en',
keyterm: ['AI SDK', 'Grok'],
streaming: {
interimResults: true,
endpointing: 500,
},
},
},
});
```
Some providers require WebSocket headers for direct streaming STT. In those runtimes, pass a provider-specific `webSocket` implementation when creating the provider.
## Settings
### Provider-Specific settings
Transcription models often have provider or model-specific settings which you can set using the `providerOptions` parameter.
```ts highlight="8-12"
import { transcribe } from 'ai';
import { openai } from '@ai-sdk/openai';
import { readFile } from 'fs/promises';
const transcript = await transcribe({
model: openai.transcription('whisper-1'),
audio: await readFile('audio.mp3'),
providerOptions: {
openai: {
timestampGranularities: ['word'],
},
},
});
```
### Download Size Limits
When `audio` is an HTTP(S) URL and the model supports native URL transcription (such as Groq), the provider fetches it directly. Provider size limits apply; the SDK does not download or inspect the audio. The URL must be accessible to the provider. Providers can decline unsupported URLs and fall back to local downloading.
Otherwise, the SDK downloads the file with a default **2 GiB** size limit.
You can customize this using `createDownload`. Supplying `download` always uses local downloading, including for models with native URL support:
```ts highlight="1,7"
import { transcribe, createDownload } from 'ai';
import { openai } from '@ai-sdk/openai';
const transcript = await transcribe({
model: openai.transcription('whisper-1'),
audio: new URL('https://example.com/audio.mp3'),
download: createDownload({ maxBytes: 50 * 1024 * 1024 }), // 50 MB limit
});
```
You can also provide a fully custom download function:
```ts highlight="7-13"
import { transcribe } from 'ai';
import { openai } from '@ai-sdk/openai';
const transcript = await transcribe({
model: openai.transcription('whisper-1'),
audio: new URL('https://example.com/audio.mp3'),
download: async ({ url }) => {
const res = await myAuthenticatedFetch(url);
return {
data: new Uint8Array(await res.arrayBuffer()),
mediaType: res.headers.get('content-type') ?? undefined,
};
},
});
```
If a download exceeds the size limit, a `DownloadError` is thrown:
```ts
import { transcribe, DownloadError } from 'ai';
import { openai } from '@ai-sdk/openai';
try {
await transcribe({
model: openai.transcription('whisper-1'),
audio: new URL('https://example.com/audio.mp3'),
});
} catch (error) {
if (DownloadError.isInstance(error)) {
console.log('Download failed:', error.message);
}
}
```
### DNS protection for audio downloads
The audio download uses the `download` option, not the provider's `fetch`. For
untrusted audio URLs on Node.js, pass a `guarded-fetch` transport via
`download: createDownload({ fetch })`. See [Secure URL
Fetching](/docs/advanced/secure-url-fetching#core-file-downloads).
### Abort Signals and Timeouts
`transcribe` accepts an optional `abortSignal` parameter of
type [`AbortSignal`](https://developer.mozilla.org/en-US/docs/Web/API/AbortSignal)
that you can use to abort the transcription process or set a timeout.
This is particularly useful when combined with URL downloads to prevent long-running requests:
```ts highlight="7"
import { openai } from '@ai-sdk/openai';
import { transcribe } from 'ai';
const transcript = await transcribe({
model: openai.transcription('whisper-1'),
audio: new URL('https://example.com/audio.mp3'),
abortSignal: AbortSignal.timeout(5000), // Abort after 5 seconds
});
```
### Custom Headers
`transcribe` accepts an optional `headers` parameter of type `Record<string, string>`
that you can use to add custom headers to the transcription request.
```ts highlight="8"
import { openai } from '@ai-sdk/openai';
import { transcribe } from 'ai';
import { readFile } from 'fs/promises';
const transcript = await transcribe({
model: openai.transcription('whisper-1'),
audio: await readFile('audio.mp3'),
headers: { 'X-Custom-Header': 'custom-value' },
});
```
### Warnings
Warnings (e.g. unsupported parameters) are available on the `warnings` property.
```ts
import { openai } from '@ai-sdk/openai';
import { transcribe } from 'ai';
import { readFile } from 'fs/promises';
const transcript = await transcribe({
model: openai.transcription('whisper-1'),
audio: await readFile('audio.mp3'),
});
const warnings = transcript.warnings;
```
### Error Handling
When `transcribe` cannot generate a valid transcript, it throws a [`AI_NoTranscriptGeneratedError`](/docs/reference/ai-sdk-errors/ai-no-transcript-generated-error).
This error can arise for any of the following reasons:
- The model failed to generate a response
- The model generated a response that could not be parsed
The error preserves the following information to help you log the issue:
- `responses`: Metadata about the transcription model responses, including timestamp, model, and headers.
- `cause`: The cause of the error. You can use this for more detailed error handling.
```ts
import { transcribe, NoTranscriptGeneratedError } from 'ai';
import { openai } from '@ai-sdk/openai';
import { readFile } from 'fs/promises';
try {
await transcribe({
model: openai.transcription('whisper-1'),
audio: await readFile('audio.mp3'),
});
} catch (error) {
if (NoTranscriptGeneratedError.isInstance(error)) {
console.log('NoTranscriptGeneratedError');
console.log('Cause:', error.cause);
console.log('Responses:', error.responses);
}
}
```
## Transcription Models
| Provider | Model |
| ----------------------------------------------------------------------------------- | ------------------------------ |
| [OpenAI](/providers/ai-sdk-providers/openai#transcription-models) | `whisper-1` |
| [OpenAI](/providers/ai-sdk-providers/openai#transcription-models) | `gpt-4o-transcribe` |
| [OpenAI](/providers/ai-sdk-providers/openai#transcription-models) | `gpt-4o-mini-transcribe` |
| [OpenAI](/providers/ai-sdk-providers/openai#transcription-models) | `gpt-4o-transcribe-diarize` |
| [ElevenLabs](/providers/ai-sdk-providers/elevenlabs#transcription-models) | `scribe_v1` |
| [ElevenLabs](/providers/ai-sdk-providers/elevenlabs#transcription-models) | `scribe_v1_experimental` |
| [ElevenLabs](/providers/ai-sdk-providers/elevenlabs#transcription-models) | `scribe_v2` |
| [ElevenLabs](/providers/ai-sdk-providers/elevenlabs#streaming-transcription-models) | `scribe_v2_realtime` |
| [Groq](/providers/ai-sdk-providers/groq#transcription-models) | `whisper-large-v3-turbo` |
| [Groq](/providers/ai-sdk-providers/groq#transcription-models) | `whisper-large-v3` |
| [Mistral](/providers/ai-sdk-providers/mistral#transcription-models) | `voxtral-mini-transcribe-3-0` |
| [Mistral](/providers/ai-sdk-providers/mistral#transcription-models) | `voxtral-small-transcribe-3-0` |
| [Azure OpenAI](/providers/ai-sdk-providers/azure#transcription-models) | `whisper-1` |
| [Azure Speech](/providers/ai-sdk-providers/azure#mai-transcribe) | `mai-transcribe-2` |
| [Azure Speech](/providers/ai-sdk-providers/azure#mai-transcribe) | `mai-transcribe-1.5` |
| [Azure MAI](/providers/ai-sdk-providers/azure#mai-transcribe-2-streaming) | `mai-transcribe-2-streaming` |
| [Azure OpenAI](/providers/ai-sdk-providers/azure#transcription-models) | `gpt-4o-transcribe` |
| [Azure OpenAI](/providers/ai-sdk-providers/azure#transcription-models) | `gpt-4o-mini-transcribe` |
| [Rev.ai](/providers/ai-sdk-providers/revai#transcription-models) | `machine` |
| [Rev.ai](/providers/ai-sdk-providers/revai#transcription-models) | `low_cost` |
| [Rev.ai](/providers/ai-sdk-providers/revai#transcription-models) | `fusion` |
| [Deepgram](/providers/ai-sdk-providers/deepgram#transcription-models) | `base` (+ variants) |
| [Deepgram](/providers/ai-sdk-providers/deepgram#transcription-models) | `enhanced` (+ variants) |
| [Deepgram](/providers/ai-sdk-providers/deepgram#transcription-models) | `nova` (+ variants) |
| [Deepgram](/providers/ai-sdk-providers/deepgram#transcription-models) | `nova-2` (+ variants) |
| [Deepgram](/providers/ai-sdk-providers/deepgram#transcription-models) | `nova-3` (+ variants) |
| [Gladia](/providers/ai-sdk-providers/gladia#transcription-models) | `default` |
| [AssemblyAI](/providers/ai-sdk-providers/assemblyai#transcription-models) | `universal-3-5-pro` |
| [AssemblyAI](/providers/ai-sdk-providers/assemblyai#transcription-models) | `universal-3-pro` |
| [Fal](/providers/ai-sdk-providers/fal#transcription-models) | `whisper` |
| [Fal](/providers/ai-sdk-providers/fal#transcription-models) | `wizper` |
| [Google Vertex](/providers/ai-sdk-providers/google-vertex#transcription-models) | `chirp_2` |
| [Google Vertex](/providers/ai-sdk-providers/google-vertex#transcription-models) | `chirp_3` |
| [Google Vertex](/providers/ai-sdk-providers/google-vertex#transcription-models) | `telephony` |
| [xAI](/providers/ai-sdk-providers/xai#transcription-models) | `default` |
| [Cartesia](/providers/ai-sdk-providers/cartesia#transcription-models) | `ink-whisper` |
| [Cartesia](/providers/ai-sdk-providers/cartesia#streaming-transcription-models) | `ink-2` |
| [Fish Audio](/providers/ai-sdk-providers/fish-audio#transcription-models) | `transcribe-1` |
Above are a small subset of the transcription models supported by the AI SDK providers. For more, see the respective provider documentation.