openclaw
Version:
Multi-channel AI gateway with extensible messaging integrations
232 lines (191 loc) • 8.57 kB
Markdown
---
summary: "Speech, media understanding, image/video/music generation, web search, and media utilities"
read_when:
- You are synthesizing speech or transcribing inbound audio
- You are generating images, video, or music from a plugin
- You need MIME detection, resizing, or QR helpers
title: "Plugin runtime media helpers"
sidebarTitle: "Media helpers"
---
Speech, media understanding, generation, web search, and the low-level media utilities. Part of the [Plugin runtime helpers](/plugins/sdk-runtime) reference.
## FFmpeg command discovery
Official plugin sources use `resolveFfmpegBin` from
the private `openclaw/plugin-sdk/media-ffmpeg` runtime. It resolves the same
trusted system paths as the media runtime and throws an installation hint when
FFmpeg is unavailable. This narrow entry point keeps media generation and agent
runtimes out of source audio worker startup. It has a JavaScript-only host export;
its declarations are excluded from the package. Third-party plugins retain the
existing `resolveFfmpegBin` export from `openclaw/plugin-sdk/media-runtime`.
The official plugin publication builder emits `media-runtime` for this private
import, including worker entries, so published plugins keep working on supported
hosts that predate `media-ffmpeg`. Both paths use the host's FFmpeg resolver.
## Realtime voice playback
Official audio workers use `createRealtimeVoiceOutputActivityTracker` and
`isRealtimeVoiceAudioAudible` from the private
`openclaw/plugin-sdk/realtime-voice-playback` runtime to avoid loading voice
session and agent runtimes from source. The publication builder emits the
existing `openclaw/plugin-sdk/realtime-voice` host import for these helpers,
including worker entries, to preserve supported hosts. The source facade has a
JavaScript-only host export; its declarations are excluded from the package.
## Media and generation namespaces
<AccordionGroup>
<Accordion title="api.runtime.tts">
Text-to-speech synthesis.
```typescript
// Standard TTS
const clip = await api.runtime.tts.textToSpeech({
text: "Hello from OpenClaw",
cfg: api.config,
});
// Telephony-optimized TTS
const telephonyClip = await api.runtime.tts.textToSpeechTelephony({
text: "Hello from OpenClaw",
cfg: api.config,
});
// List available voices
const voices = await api.runtime.tts.listVoices({
provider: "elevenlabs",
cfg: api.config,
});
```
Uses core `tts` configuration and provider selection. Returns PCM audio buffer + sample rate. `textToSpeechStream` is also available for streaming synthesis.
</Accordion>
<Accordion title="api.runtime.mediaUnderstanding">
Image, audio, and video analysis.
```typescript
// Describe an image
const image = await api.runtime.mediaUnderstanding.describeImageFile({
filePath: "/tmp/inbound-photo.jpg",
cfg: api.config,
agentDir: "/tmp/agent",
});
// Prepare a capture limit before installing audio receive listeners.
const budget = await api.runtime.mediaUnderstanding.resolveAudioInputBudget({
cfg: api.config,
});
// budget.enabled is false when audio understanding is disabled. Otherwise,
// budget.maxBytes includes the container header and covers the largest fallback.
// Transcribe audio
const { text } = await api.runtime.mediaUnderstanding.transcribeAudioFile({
filePath: "/tmp/inbound-audio.ogg",
cfg: api.config,
mime: "audio/ogg", // optional, for when MIME cannot be inferred
});
// Describe a video
const video = await api.runtime.mediaUnderstanding.describeVideoFile({
filePath: "/tmp/inbound-video.mp4",
cfg: api.config,
});
// Generic file analysis
const result = await api.runtime.mediaUnderstanding.runFile({
filePath: "/tmp/inbound-file.pdf",
cfg: api.config,
});
// Structured image extraction through a specific provider/model.
// Include at least one image; text inputs are supplemental context.
// receiptImageBuffer is your own image bytes, not an SDK-provided value.
const evidence = await api.runtime.mediaUnderstanding.extractStructuredWithModel({
provider: "codex",
model: "gpt-6-astra",
input: [
{
type: "image",
buffer: receiptImageBuffer,
fileName: "receipt.png",
mime: "image/png",
},
{ type: "text", text: "Prefer the printed total over handwritten notes." },
],
instructions: "Extract vendor, total, and searchable tags.",
schemaName: "receipt.evidence",
jsonSchema: {
type: "object",
properties: {
vendor: { type: "string" },
total: { type: "number" },
tags: { type: "array", items: { type: "string" } },
},
required: ["vendor", "total"],
},
cfg: api.config,
});
```
Returns `{ text: undefined }` when no output is produced (e.g. skipped input).
`describeImageFileWithModel(...)` describes an already-known image through a specific provider/model, bypassing the default active-model resolution that `describeImageFile(...)` uses.
</Accordion>
<Accordion title="api.runtime.imageGeneration">
Image generation.
```typescript
const result = await api.runtime.imageGeneration.generate({
prompt: "A robot painting a sunset",
cfg: api.config,
});
const providers = api.runtime.imageGeneration.listProviders({ cfg: api.config });
```
</Accordion>
<Accordion title="api.runtime.videoGeneration">
Video generation, mirroring the image generation shape.
```typescript
const result = await api.runtime.videoGeneration.generate({
prompt: "A drone shot flying over a coastline at sunrise",
cfg: api.config,
});
const providers = api.runtime.videoGeneration.listProviders({ cfg: api.config });
```
</Accordion>
<Accordion title="api.runtime.musicGeneration">
Music generation, mirroring the image generation shape.
```typescript
const result = await api.runtime.musicGeneration.generate({
prompt: "An upbeat lo-fi track for a coding session",
cfg: api.config,
});
const providers = api.runtime.musicGeneration.listProviders({ cfg: api.config });
```
</Accordion>
<Accordion title="api.runtime.webSearch">
Web search.
```typescript
const providers = api.runtime.webSearch.listProviders({ config: api.config });
const result = await api.runtime.webSearch.search({
config: api.config,
args: { query: "OpenClaw plugin SDK", count: 5 },
});
```
Search callers may supply a synchronous `assertCurrent` callback with `signal`
to retain their authority through provider preparation. Guarded HTTP requests
check it after transport preparation and before each request or redirect.
A registered search provider using another transport must call the execution
context's `assertCurrent` before each external side effect, after awaited
preparation. The callback belongs to the caller and expires when the search
finishes; providers must not replace it or treat its absence as permission.
</Accordion>
<Accordion title="api.runtime.media">
Low-level media utilities.
```typescript
const webMedia = await api.runtime.media.loadWebMedia(url);
const mime = await api.runtime.media.detectMime({ buffer });
const kind = api.runtime.media.mediaKindFromMime("image/jpeg"); // "image"
const isVoice = api.runtime.media.isVoiceCompatibleAudio({ fileName: filePath });
const metadata = await api.runtime.media.getImageMetadata(buffer);
const resized = await api.runtime.media.resizeToJpeg({ buffer, maxSide: 800, quality: 85 });
```
QR helpers are exported by `openclaw/plugin-sdk/media-runtime`:
```typescript
import { resolvePreferredOpenClawTmpDir } from "openclaw/plugin-sdk/temp-path";
const qr = await import("openclaw/plugin-sdk/media-runtime");
const terminalQr = await qr.renderQrTerminal("https://openclaw.ai");
const pngQr = await qr.renderQrPngBase64("https://openclaw.ai", {
scale: 6, // 1-12
marginModules: 4, // 0-16
});
const pngQrDataUrl = await qr.renderQrPngDataUrl("https://openclaw.ai");
const tmpRoot = resolvePreferredOpenClawTmpDir();
const pngQrFile = await qr.writeQrPngTempFile("https://openclaw.ai", {
tmpRoot,
dirPrefix: "my-plugin-qr-",
fileName: "qr.png",
});
```
</Accordion>
</AccordionGroup>