UNPKG

sandai-react

Version:

React components and utilities for the Sandai 3D AI Characters.

245 lines (235 loc) 40.3 kB
<!DOCTYPE html><html class="default" lang="en" data-base="."><head><meta charset="utf-8"/><meta http-equiv="x-ua-compatible" content="IE=edge"/><title>sandai-react - v2.1.8</title><meta name="description" content="Documentation for sandai-react"/><meta name="viewport" content="width=device-width, initial-scale=1"/><link rel="stylesheet" href="assets/style.css"/><link rel="stylesheet" href="assets/highlight.css"/><script defer src="assets/main.js"></script><script async src="assets/icons.js" id="tsd-icons-script"></script><script async src="assets/search.js" id="tsd-search-script"></script><script async src="assets/navigation.js" id="tsd-nav-script"></script></head><body><script>document.documentElement.dataset.theme = localStorage.getItem("tsd-theme") || "os";document.body.style.display="none";setTimeout(() => app?app.showPage():document.body.style.removeProperty("display"),500)</script><header class="tsd-page-toolbar"><div class="tsd-toolbar-contents container"><div class="table-cell" id="tsd-search"><div class="field"><label for="tsd-search-field" class="tsd-widget tsd-toolbar-icon search no-caption"><svg width="16" height="16" viewBox="0 0 16 16" fill="none" aria-hidden="true"><use href="assets/icons.svg#icon-search"></use></svg></label><input type="text" id="tsd-search-field" aria-label="Search"/></div><div class="field"><div id="tsd-toolbar-links"></div></div><ul class="results"><li class="state loading">Preparing search index...</li><li class="state failure">The search index is not available</li></ul><a href="index.html" class="title">sandai-react - v2.1.8</a></div><div class="table-cell" id="tsd-widgets"><a href="#" class="tsd-widget tsd-toolbar-icon menu no-caption" data-toggle="menu" aria-label="Menu"><svg width="16" height="16" viewBox="0 0 16 16" fill="none" aria-hidden="true"><use href="assets/icons.svg#icon-menu"></use></svg></a></div></div></header><div class="container container-main"><div class="col-content"><div class="tsd-page-title"><h1>sandai-react - v2.1.8</h1></div><div class="tsd-panel tsd-typography"><a id="sandai-react" class="tsd-anchor"></a><h1 class="tsd-anchor-link">sandai-react<a href="#sandai-react" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h1><p><strong>The React way to embed a conversational 3D AI avatar; on-device, privacy-first, one component.</strong></p> <p>See what it's able to do in a <a href="https://sustainabilitymap-mit-horst.diekavallerie.de/"><strong>custom implementation</strong> from <strong>DIE KAVALLERIE</strong> 🐴</a></p> <p><strong>Live playground:</strong> <a href="https://sandai.org/api_playground">sandai.org</a></p> <p><code>sandai-react</code> is the React SDK for dropping a fully interactive <strong>3D AI avatar</strong> into any React or Next.js app. Mount a single <code>&lt;AI3DCharacter /&gt;</code> component and you get a real-time <strong>AI avatar</strong>. A browser-based <strong>digital human, mascot, animal, you name it</strong>, that can speak, listen, think, and emote. Under the hood it renders the same heavily optimized runtime as <a href="https://gitlab.com/DavidCks1/sandai-core"><code>sandai-core</code></a>, so you get a complete <strong>conversational AI avatar</strong>: on-device <strong>text-to-speech</strong>, <strong>speech recognition</strong>, an <strong>on-device LLM</strong>, <strong>emotion inference</strong>, and <strong>lip-sync</strong>, with no backend to run and nothing to install.</p> <p>By default, <strong>everything runs locally in the browser</strong> via WebAssembly and WebGL/WebGPU. No API keys, no per-message costs, and no user audio ever leaves the device.</p> <p><code>sandai-react</code> re-exports all of <code>sandai-core</code> and wraps it in idiomatic React: a declarative component that builds the embed URL from props and hands you a typed <code>SandaiClient</code> in <code>onLoad</code>. It builds on the same stack: <a href="https://gitlab.com/DavidCks1/sandai-core"><code>sandai-core</code></a><a href="https://gitlab.com/DavidCks1/ai-vrm-chat"><code>ai-vrm-chat</code></a><a href="https://gitlab.com/DavidCks1/ai-character"><code>ai-character</code></a><a href="https://gitlab.com/DavidCks1/r3f-vrm"><code>@davidcks/r3f-vrm</code></a> (Three.js / React Three Fiber).</p> <a id="features" class="tsd-anchor"></a><h2 class="tsd-anchor-link">Features<a href="#features" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h2><ul> <li><strong>One component, zero boilerplate</strong>: render <code>&lt;AI3DCharacter /&gt;</code>, get a typed <code>SandaiClient</code> in <code>onLoad</code>. No iframe wiring, no <code>postMessage</code> plumbing, no 3D engine, no game engine, no local GPU server, no API gateway.</li> <li><strong>3D VRM avatars</strong>: load any <code>.vrm</code> model and get a fully animated <strong>talking avatar</strong> with auto-blink, gaze, idle motion, and camera focus. Bring your own character, use a bundled one, or <strong>pick from over 100,000 models</strong> at <a href="https://sandai.org/dashboard#API-Playground">sandai.org</a>.</li> <li><strong>On-device LLM</strong>: a <strong>local LLM</strong> (Gemma, Qwen, and others via transformers.js / ONNX and MediaPipe LiteRT-LM) generates replies in-browser. Or plug in your own and just stream text in.</li> <li><strong>Text-to-speech, 100+ voices, 30+ languages</strong>: built-in multilingual <strong>TTS</strong> powered by Piper, selectable per character with the typed <code>voiceName</code> prop.</li> <li><strong>Real-time lip-sync</strong>: accurate <strong>lip-sync</strong> driven from phonemes <em>or</em> directly from the audio waveform, with coarticulation so the mouth never looks robotic.</li> <li><strong>Speech recognition (ASR) + VAD</strong>: the avatar can <strong>listen</strong> and reply on its own via on-device Whisper / Moonshine speech-to-text with Silero voice-activity detection.</li> <li><strong>Emotion inference &amp; expression</strong>: text is mapped to 28 GoEmotions-style <strong>emotions</strong> that automatically drive <strong>facial expressions</strong> and graded <strong>body animations</strong>.</li> <li><strong>Fully typed</strong>: props, voices (<code>VoiceNames</code>), environments (<code>Environments</code>), and the entire <code>SandaiClient</code> RPC surface ship with TypeScript types and autocomplete.</li> <li><strong>WebGL &amp; WebGPU rendering</strong>, with an optional experimental GPU path-tracer (<code>raytrace</code>) for cinematic, ray-traced scenes.</li> <li><strong>Built-in dev tooling</strong>: <code>AI3DCharacterDocs</code> renders interactive, runnable documentation for every method the character exposes — a live playground inside your own app.</li> <li><strong>Privacy-first, no backend</strong>: conversation, voice, and inference happen on the user's device. Great for GDPR-sensitive products, offline-capable apps, and <strong>keeping inference costs at zero</strong>.</li> </ul> <a id="use-cases" class="tsd-anchor"></a><h2 class="tsd-anchor-link">Use cases<a href="#use-cases" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h2><p><strong>AI agents</strong> and website assistants, customer support and onboarding, <strong>AI companions</strong> and VTubers, interactive characters for games and education, <strong>AI mascots</strong> and brand mascots and virtual presenters, and any React product that benefits from a face you can talk to.</p> <a id="table-of-contents" class="tsd-anchor"></a><h2 class="tsd-anchor-link">Table of Contents<a href="#table-of-contents" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h2><ul> <li><a href="#sandai-react">sandai-react</a> <ul> <li><a href="#features">Features</a></li> <li><a href="#use-cases">Use cases</a></li> <li><a href="#table-of-contents">Table of Contents</a></li> <li><a href="#installation">Installation</a></li> <li><a href="#quick-start">Quick Start</a></li> <li><a href="#components">Components</a> <ul> <li><a href="#ai3dcharacter"><code>AI3DCharacter</code></a></li> <li><a href="#ai3dcharacterdocs"><code>AI3DCharacterDocs</code></a></li> <li><a href="#functiontester"><code>FunctionTester</code></a></li> </ul> </li> <li><a href="#driving-the-character">Driving the character</a></li> <li><a href="#re-exports-from-sandai-core">Re-exports from sandai-core</a></li> <li><a href="#contributing">Contributing</a></li> <li><a href="#license">License</a></li> </ul> </li> </ul> <a id="installation" class="tsd-anchor"></a><h2 class="tsd-anchor-link">Installation<a href="#installation" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h2><pre><code class="sh"><span class="hl-0">npm</span><span class="hl-1"> </span><span class="hl-2">install</span><span class="hl-1"> </span><span class="hl-2">sandai-react</span> </code><button type="button">Copy</button></pre> <p>Peer dependencies (React 19):</p> <pre><code class="sh"><span class="hl-0">npm</span><span class="hl-1"> </span><span class="hl-2">install</span><span class="hl-1"> </span><span class="hl-2">react</span><span class="hl-1"> </span><span class="hl-2">react-dom</span> </code><button type="button">Copy</button></pre> <a id="quick-start" class="tsd-anchor"></a><h2 class="tsd-anchor-link">Quick Start<a href="#quick-start" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h2><p>Mount the component, size its container, and grab the client in <code>onLoad</code>:</p> <pre><code class="tsx"><span class="hl-3">import</span><span class="hl-1"> { </span><span class="hl-4">AI3DCharacter</span><span class="hl-1">, </span><span class="hl-4">SandaiClient</span><span class="hl-1"> } </span><span class="hl-3">from</span><span class="hl-1"> </span><span class="hl-2">&quot;sandai-react&quot;</span><span class="hl-1">;</span><br/><span class="hl-3">import</span><span class="hl-1"> { </span><span class="hl-4">useState</span><span class="hl-1"> } </span><span class="hl-3">from</span><span class="hl-1"> </span><span class="hl-2">&quot;react&quot;</span><span class="hl-1">;</span><br/><br/><span class="hl-3">export</span><span class="hl-1"> </span><span class="hl-3">default</span><span class="hl-1"> </span><span class="hl-5">function</span><span class="hl-1"> </span><span class="hl-0">App</span><span class="hl-1">() {</span><br/><span class="hl-1"> </span><span class="hl-5">const</span><span class="hl-1"> [</span><span class="hl-6">client</span><span class="hl-1">, </span><span class="hl-6">setClient</span><span class="hl-1">] = </span><span class="hl-0">useState</span><span class="hl-1">&lt;</span><span class="hl-7">SandaiClient</span><span class="hl-1">&gt;();</span><br/><br/><span class="hl-1"> </span><span class="hl-3">return</span><span class="hl-1"> (</span><br/><span class="hl-1"> </span><span class="hl-8">&lt;</span><span class="hl-9">div</span><span class="hl-1"> </span><span class="hl-10">style</span><span class="hl-1">=</span><span class="hl-5">{</span><span class="hl-11">{ </span><span class="hl-4">width:</span><span class="hl-11"> </span><span class="hl-2">&quot;100%&quot;</span><span class="hl-11">, </span><span class="hl-4">height:</span><span class="hl-11"> </span><span class="hl-2">&quot;600px&quot;</span><span class="hl-11"> }</span><span class="hl-5">}</span><span class="hl-8">&gt;</span><br/><span class="hl-1"> </span><span class="hl-8">&lt;</span><span class="hl-7">AI3DCharacter</span><br/><span class="hl-1"> </span><span class="hl-10">url</span><span class="hl-1">=</span><span class="hl-2">&quot;https://sandai.org/chat&quot;</span><br/><span class="hl-1"> </span><span class="hl-10">vrmUrl</span><span class="hl-1">=</span><span class="hl-2">&quot;https://cdn.example/ada.vrm&quot;</span><br/><span class="hl-1"> </span><span class="hl-10">voiceName</span><span class="hl-1">=</span><span class="hl-2">&quot;ruri&quot;</span><br/><span class="hl-1"> </span><span class="hl-10">environment</span><span class="hl-1">=</span><span class="hl-2">&quot;tokyo&quot;</span><br/><span class="hl-1"> </span><span class="hl-12">// userId &amp; apiKey are optional. Without them you may see a watermark.</span><br/><span class="hl-1"> </span><span class="hl-12">// Grab yours in the dashboard: https://sandai.org/dashboard</span><br/><span class="hl-1"> </span><span class="hl-10">onLoad</span><span class="hl-1">=</span><span class="hl-5">{async</span><span class="hl-11"> (</span><span class="hl-4">c</span><span class="hl-11">) </span><span class="hl-5">=&gt;</span><span class="hl-11"> {</span><br/><span class="hl-11"> </span><span class="hl-0">setClient</span><span class="hl-11">(</span><span class="hl-4">c</span><span class="hl-11">);</span><br/><span class="hl-11"> </span><span class="hl-12">// Speak some text (emotion + lip-sync inferred automatically):</span><br/><span class="hl-11"> </span><span class="hl-5">const</span><span class="hl-11"> { </span><span class="hl-6">speechEndPromise</span><span class="hl-11"> } </span><span class="hl-1">=</span><span class="hl-11"> </span><span class="hl-3">await</span><span class="hl-11"> </span><span class="hl-4">c</span><span class="hl-11">.</span><span class="hl-4">interactionManager</span><span class="hl-11">.</span><span class="hl-0">say</span><span class="hl-11">(</span><span class="hl-2">&quot;Hi there!&quot;</span><span class="hl-11">);</span><br/><span class="hl-11"> </span><span class="hl-3">await</span><span class="hl-11"> </span><span class="hl-4">speechEndPromise</span><span class="hl-11">;</span><br/><span class="hl-11"> </span><span class="hl-12">// Or have the on-device LLM reply:</span><br/><span class="hl-11"> </span><span class="hl-3">await</span><span class="hl-11"> </span><span class="hl-4">c</span><span class="hl-11">.</span><span class="hl-4">interactionManager</span><span class="hl-11">.</span><span class="hl-0">respond</span><span class="hl-11">(</span><span class="hl-2">&quot;What&#39;s your name?&quot;</span><span class="hl-11">);</span><br/><span class="hl-11"> }</span><span class="hl-5">}</span><br/><span class="hl-1"> </span><span class="hl-8">/&gt;</span><br/><span class="hl-1"> </span><span class="hl-8">&lt;/</span><span class="hl-9">div</span><span class="hl-8">&gt;</span><br/><span class="hl-1"> );</span><br/><span class="hl-1">}</span> </code><button type="button">Copy</button></pre> <blockquote> <p>The component renders an <code>iframe</code> at <code>width: 100%; height: 100%</code>, so give its parent an explicit size.</p> <p>Audio can't play until the user has interacted with the page (browser autoplay policy). If the character won't speak, that's almost always why — the underlying <code>LoadManager</code> tracks interaction state for you.</p> </blockquote> <a id="components" class="tsd-anchor"></a><h2 class="tsd-anchor-link">Components<a href="#components" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h2><a id="ai3dcharacter" class="tsd-anchor"></a><h3 class="tsd-anchor-link"><code>AI3DCharacter</code><a href="#ai3dcharacter" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h3><p>The one component you mount. It builds the Sandai embed URL from its props, renders the avatar in an <code>iframe</code>, constructs a <a href="https://gitlab.com/DavidCks1/sandai-core"><code>SandaiClient</code></a>, and hands it to you via <code>onLoad</code> once it's ready.</p> <pre><code class="tsx"><span class="hl-3">import</span><span class="hl-1"> { </span><span class="hl-4">AI3DCharacter</span><span class="hl-1"> } </span><span class="hl-3">from</span><span class="hl-1"> </span><span class="hl-2">&quot;sandai-react&quot;</span><span class="hl-1">;</span> </code><button type="button">Copy</button></pre> <a id="props" class="tsd-anchor"></a><h4 class="tsd-anchor-link">Props<a href="#props" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h4><table> <thead> <tr> <th>Prop</th> <th>Type</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code>url</code></td> <td><code>string</code></td> <td><strong>Required.</strong> Base URL of the Sandai chat interface (e.g. <code>https://sandai.org/chat</code>).</td> </tr> <tr> <td><code>onLoad</code></td> <td><code>(client: SandaiClient) =&gt; void</code></td> <td><strong>Required.</strong> Fired once the <code>SandaiClient</code> is initialized and ready to drive.</td> </tr> <tr> <td><code>vrmUrl</code></td> <td><code>string</code></td> <td>URL of the <code>.vrm</code> model to load as the character.</td> </tr> <tr> <td><code>voiceName</code></td> <td><code>VoiceNames</code></td> <td>TTS voice key (100+ multilingual Piper voices).</td> </tr> <tr> <td><code>environment</code></td> <td><code>Environments</code></td> <td>Environment preset (e.g. <code>transparent</code>, <code>tokyo</code>, <code>sky</code>, <code>candyshop</code>, ...).</td> </tr> <tr> <td><code>showControls</code></td> <td><code>boolean</code></td> <td>Show the debug overlay: chat input, emotion sliders, and built-in voices. Handy for previewing how the 5 VRM blendshapes combine into Sandai's 27 emotions.</td> </tr> <tr> <td><code>lowPerformanceMode</code></td> <td><code>boolean</code></td> <td>Lighter render path (drops bloom/depth-of-field) for older or low-end devices.</td> </tr> <tr> <td><code>cameraType</code></td> <td><code>&quot;default&quot; | &quot;orthographic&quot;</code></td> <td>Camera projection. <code>default</code> is perspective.</td> </tr> <tr> <td><code>initialFocus</code></td> <td><code>FocusProps</code></td> <td>Initial camera focus (<code>focusIntensity</code>, <code>cameraOffset</code>, <code>lookAtOffset</code>, <code>trackCharacterLookAt</code>, <code>focusMode</code>).</td> </tr> <tr> <td><code>raytrace</code></td> <td><code>boolean</code></td> <td>Enable the experimental GPU path-tracer.</td> </tr> <tr> <td><code>headAnimationSmoothing</code></td> <td><code>number</code></td> <td>Dampen head motion, <code>0..1</code> (0 = raw, 1 = frozen).</td> </tr> <tr> <td><code>bodyAnimationSmoothing</code></td> <td><code>number</code></td> <td>Dampen body motion, <code>0..1</code>.</td> </tr> <tr> <td><code>armsAnimationSmoothing</code></td> <td><code>number</code></td> <td>Dampen arm motion, <code>0..1</code>.</td> </tr> <tr> <td><code>legsAnimationSmoothing</code></td> <td><code>number</code></td> <td>Dampen leg motion, <code>0..1</code>.</td> </tr> <tr> <td><code>handsAnimationSmoothing</code></td> <td><code>number</code></td> <td>Dampen hand motion, <code>0..1</code>.</td> </tr> <tr> <td><code>faceAnimationSmoothing</code></td> <td><code>number</code></td> <td>Dampen face/expression motion, <code>0..1</code>. Try <code>0.1</code><code>0.9</code> if the mouth looks choppy.</td> </tr> <tr> <td><code>decoupleOverlappingExpressionsExperimental</code></td> <td><code>boolean</code></td> <td>Best-guess split of conflicting blendshapes (e.g. a model that opens its mouth when <em>happy</em>).</td> </tr> <tr> <td><code>apiKey</code></td> <td><code>string</code></td> <td>Your API key (from the <a href="https://sandai.org/dashboard">dashboard</a>). Optional; omit and you may see a watermark.</td> </tr> <tr> <td><code>userId</code></td> <td><code>string</code></td> <td>Your user ID (from the dashboard).</td> </tr> <tr> <td><code>debugMode</code></td> <td><code>boolean</code></td> <td>Skips auth + load checks. Convenient locally; <strong>turn off in production</strong> (it disables the interaction gate used to enable audio).</td> </tr> <tr> <td><code>style</code></td> <td><code>React.CSSProperties</code></td> <td>Inline styles for the underlying <code>iframe</code>.</td> </tr> <tr> <td><code>className</code></td> <td><code>string</code></td> <td>Class name for the underlying <code>iframe</code>.</td> </tr> </tbody> </table> <a id="example" class="tsd-anchor"></a><h4 class="tsd-anchor-link">Example<a href="#example" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h4><pre><code class="tsx"><span class="hl-3">import</span><span class="hl-1"> { </span><span class="hl-4">AI3DCharacter</span><span class="hl-1"> } </span><span class="hl-3">from</span><span class="hl-1"> </span><span class="hl-2">&quot;sandai-react&quot;</span><span class="hl-1">;</span><br/><br/><span class="hl-8">&lt;</span><span class="hl-7">AI3DCharacter</span><br/><span class="hl-1"> </span><span class="hl-10">url</span><span class="hl-1">=</span><span class="hl-2">&quot;https://sandai.org/chat&quot;</span><br/><span class="hl-1"> </span><span class="hl-10">vrmUrl</span><span class="hl-1">=</span><span class="hl-2">&quot;https://example.com/model.vrm&quot;</span><br/><span class="hl-1"> </span><span class="hl-10">voiceName</span><span class="hl-1">=</span><span class="hl-2">&quot;ruri&quot;</span><br/><span class="hl-1"> </span><span class="hl-10">environment</span><span class="hl-1">=</span><span class="hl-2">&quot;tokyo&quot;</span><br/><span class="hl-1"> </span><span class="hl-10">showControls</span><br/><span class="hl-1"> </span><span class="hl-10">faceAnimationSmoothing</span><span class="hl-1">=</span><span class="hl-5">{</span><span class="hl-13">0.4</span><span class="hl-5">}</span><br/><span class="hl-1"> </span><span class="hl-10">onLoad</span><span class="hl-1">=</span><span class="hl-5">{</span><span class="hl-11">(</span><span class="hl-4">client</span><span class="hl-11">) </span><span class="hl-5">=&gt;</span><span class="hl-11"> </span><span class="hl-4">console</span><span class="hl-11">.</span><span class="hl-0">log</span><span class="hl-11">(</span><span class="hl-2">&quot;Sandai client ready&quot;</span><span class="hl-11">, </span><span class="hl-4">client</span><span class="hl-11">)</span><span class="hl-5">}</span><br/><span class="hl-8">/&gt;</span> </code><button type="button">Copy</button></pre> <hr> <a id="ai3dcharacterdocs" class="tsd-anchor"></a><h3 class="tsd-anchor-link"><code>AI3DCharacterDocs</code><a href="#ai3dcharacterdocs" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h3><p>Renders <strong>interactive, runnable documentation</strong> for every method the loaded <code>SandaiClient</code> exposes — a live API playground you can drop straight into your own app. Pass it the <code>client</code> from <code>AI3DCharacter</code>'s <code>onLoad</code>.</p> <a id="props-1" class="tsd-anchor"></a><h4 class="tsd-anchor-link">Props<a href="#props-1" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h4><table> <thead> <tr> <th>Prop</th> <th>Type</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code>client</code></td> <td><code>SandaiClient</code></td> <td>The instance from <code>AI3DCharacter</code>'s <code>onLoad</code> callback.</td> </tr> </tbody> </table> <a id="example-1" class="tsd-anchor"></a><h4 class="tsd-anchor-link">Example<a href="#example-1" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h4><pre><code class="tsx"><span class="hl-3">import</span><span class="hl-1"> { </span><span class="hl-4">AI3DCharacter</span><span class="hl-1">, </span><span class="hl-4">AI3DCharacterDocs</span><span class="hl-1"> } </span><span class="hl-3">from</span><span class="hl-1"> </span><span class="hl-2">&quot;sandai-react&quot;</span><span class="hl-1">;</span><br/><span class="hl-3">import</span><span class="hl-1"> { </span><span class="hl-4">useState</span><span class="hl-1"> } </span><span class="hl-3">from</span><span class="hl-1"> </span><span class="hl-2">&quot;react&quot;</span><span class="hl-1">;</span><br/><br/><span class="hl-5">const</span><span class="hl-1"> </span><span class="hl-0">App</span><span class="hl-1"> = () </span><span class="hl-5">=&gt;</span><span class="hl-1"> {</span><br/><span class="hl-1"> </span><span class="hl-5">const</span><span class="hl-1"> [</span><span class="hl-6">client</span><span class="hl-1">, </span><span class="hl-6">setClient</span><span class="hl-1">] = </span><span class="hl-0">useState</span><span class="hl-1">(</span><span class="hl-5">null</span><span class="hl-1">);</span><br/><br/><span class="hl-1"> </span><span class="hl-3">return</span><span class="hl-1"> (</span><br/><span class="hl-1"> </span><span class="hl-8">&lt;&gt;</span><br/><span class="hl-1"> </span><span class="hl-8">&lt;</span><span class="hl-7">AI3DCharacter</span><span class="hl-1"> </span><span class="hl-10">url</span><span class="hl-1">=</span><span class="hl-2">&quot;https://sandai.org/chat&quot;</span><span class="hl-1"> </span><span class="hl-10">onLoad</span><span class="hl-1">=</span><span class="hl-5">{</span><span class="hl-4">setClient</span><span class="hl-5">}</span><span class="hl-1"> </span><span class="hl-10">showControls</span><span class="hl-1"> </span><span class="hl-8">/&gt;</span><br/><span class="hl-1"> </span><span class="hl-5">{</span><span class="hl-4">client</span><span class="hl-11"> </span><span class="hl-1">&amp;&amp;</span><span class="hl-11"> </span><span class="hl-8">&lt;</span><span class="hl-7">AI3DCharacterDocs</span><span class="hl-11"> </span><span class="hl-10">client</span><span class="hl-1">=</span><span class="hl-5">{</span><span class="hl-4">client</span><span class="hl-5">}</span><span class="hl-11"> </span><span class="hl-8">/&gt;</span><span class="hl-5">}</span><br/><span class="hl-1"> </span><span class="hl-8">&lt;/&gt;</span><br/><span class="hl-1"> );</span><br/><span class="hl-1">};</span> </code><button type="button">Copy</button></pre> <hr> <a id="functiontester" class="tsd-anchor"></a><h3 class="tsd-anchor-link"><code>FunctionTester</code><a href="#functiontester" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h3><p>A reusable form that dynamically generates input fields from a schema and calls a function on submit. It powers <code>AI3DCharacterDocs</code>, and you can use it standalone to build quick test harnesses around any function.</p> <a id="props-2" class="tsd-anchor"></a><h4 class="tsd-anchor-link">Props<a href="#props-2" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h4><table> <thead> <tr> <th>Prop</th> <th>Type</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td><code>func</code></td> <td><code>Function</code></td> <td>The function to invoke; receives the form values as arguments.</td> </tr> <tr> <td><code>schema</code></td> <td><code>SchemaType[]</code></td> <td>Field definitions for the function parameters.</td> </tr> </tbody> </table> <a id="example-2" class="tsd-anchor"></a><h4 class="tsd-anchor-link">Example<a href="#example-2" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h4><pre><code class="tsx"><span class="hl-3">import</span><span class="hl-1"> { </span><span class="hl-4">FunctionTester</span><span class="hl-1"> } </span><span class="hl-3">from</span><span class="hl-1"> </span><span class="hl-2">&quot;sandai-react&quot;</span><span class="hl-1">;</span><br/><br/><span class="hl-5">const</span><span class="hl-1"> </span><span class="hl-0">myFunction</span><span class="hl-1"> = (</span><span class="hl-4">name</span><span class="hl-1">, </span><span class="hl-4">age</span><span class="hl-1">) </span><span class="hl-5">=&gt;</span><span class="hl-1"> </span><span class="hl-4">console</span><span class="hl-1">.</span><span class="hl-0">log</span><span class="hl-1">(</span><span class="hl-2">`Name: </span><span class="hl-5">${</span><span class="hl-4">name</span><span class="hl-5">}</span><span class="hl-2">, Age: </span><span class="hl-5">${</span><span class="hl-4">age</span><span class="hl-5">}</span><span class="hl-2">`</span><span class="hl-1">);</span><br/><br/><span class="hl-5">const</span><span class="hl-1"> </span><span class="hl-6">schema</span><span class="hl-1"> = [</span><br/><span class="hl-1"> { </span><span class="hl-4">id:</span><span class="hl-1"> </span><span class="hl-2">&quot;name&quot;</span><span class="hl-1">, </span><span class="hl-4">type:</span><span class="hl-1"> </span><span class="hl-2">&quot;text&quot;</span><span class="hl-1">, </span><span class="hl-4">label:</span><span class="hl-1"> </span><span class="hl-2">&quot;Name&quot;</span><span class="hl-1"> },</span><br/><span class="hl-1"> { </span><span class="hl-4">id:</span><span class="hl-1"> </span><span class="hl-2">&quot;age&quot;</span><span class="hl-1">, </span><span class="hl-4">type:</span><span class="hl-1"> </span><span class="hl-2">&quot;number&quot;</span><span class="hl-1">, </span><span class="hl-4">label:</span><span class="hl-1"> </span><span class="hl-2">&quot;Age&quot;</span><span class="hl-1"> },</span><br/><span class="hl-1">];</span><br/><br/><span class="hl-8">&lt;</span><span class="hl-7">FunctionTester</span><span class="hl-1"> </span><span class="hl-10">func</span><span class="hl-1">=</span><span class="hl-5">{</span><span class="hl-4">myFunction</span><span class="hl-5">}</span><span class="hl-1"> </span><span class="hl-10">schema</span><span class="hl-1">=</span><span class="hl-5">{</span><span class="hl-4">schema</span><span class="hl-5">}</span><span class="hl-1"> </span><span class="hl-8">/&gt;</span><span class="hl-1">;</span> </code><button type="button">Copy</button></pre> <a id="driving-the-character" class="tsd-anchor"></a><h2 class="tsd-anchor-link">Driving the character<a href="#driving-the-character" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h2><p><code>onLoad</code> hands you a <code>SandaiClient</code>, which is your control surface for the avatar. The most common calls:</p> <pre><code class="tsx"><span class="hl-12">// Speak text — emotion + lip-sync are inferred automatically.</span><br/><span class="hl-5">const</span><span class="hl-1"> { </span><span class="hl-6">speechEndPromise</span><span class="hl-1"> } = </span><span class="hl-3">await</span><span class="hl-1"> </span><span class="hl-4">client</span><span class="hl-1">.</span><span class="hl-4">interactionManager</span><span class="hl-1">.</span><span class="hl-0">say</span><span class="hl-1">(</span><span class="hl-2">&quot;Hello!&quot;</span><span class="hl-1">);</span><br/><span class="hl-3">await</span><span class="hl-1"> </span><span class="hl-4">speechEndPromise</span><span class="hl-1">;</span><br/><br/><span class="hl-12">// Ask the on-device LLM for a reply, then speak it.</span><br/><span class="hl-5">const</span><span class="hl-1"> </span><span class="hl-6">reply</span><span class="hl-1"> = </span><span class="hl-3">await</span><span class="hl-1"> </span><span class="hl-4">client</span><span class="hl-1">.</span><span class="hl-4">interactionManager</span><span class="hl-1">.</span><span class="hl-0">respond</span><span class="hl-1">(</span><span class="hl-2">&quot;Tell me a joke about space.&quot;</span><span class="hl-1">);</span><br/><br/><span class="hl-12">// Interrupt the current utterance.</span><br/><span class="hl-3">await</span><span class="hl-1"> </span><span class="hl-4">client</span><span class="hl-1">.</span><span class="hl-4">interactionManager</span><span class="hl-1">.</span><span class="hl-0">stop</span><span class="hl-1">();</span><br/><br/><span class="hl-12">// Call any method on the character by typed dot-path (full autocomplete).</span><br/><span class="hl-3">await</span><span class="hl-1"> </span><span class="hl-4">client</span><span class="hl-1">.</span><span class="hl-4">rpcManager</span><span class="hl-1">.</span><span class="hl-0">call</span><span class="hl-1">(</span><span class="hl-2">&quot;setEmotion&quot;</span><span class="hl-1">, </span><span class="hl-2">&quot;joy&quot;</span><span class="hl-1">, </span><span class="hl-13">0.8</span><span class="hl-1">);</span><br/><span class="hl-3">await</span><span class="hl-1"> </span><span class="hl-4">client</span><span class="hl-1">.</span><span class="hl-4">rpcManager</span><span class="hl-1">.</span><span class="hl-0">call</span><span class="hl-1">(</span><span class="hl-2">&quot;vrmManager.focusManager.focus&quot;</span><span class="hl-1">);</span> </code><button type="button">Copy</button></pre> <p>The full <code>SandaiClient</code> API — <code>interactionManager</code>, <code>loadManager</code>, <code>authManager</code>, and the typed <code>rpcManager</code> — is documented in the <a href="https://gitlab.com/DavidCks1/sandai-core"><code>sandai-core</code> README</a>.</p> <a id="re-exports-from-sandai-core" class="tsd-anchor"></a><h2 class="tsd-anchor-link">Re-exports from sandai-core<a href="#re-exports-from-sandai-core" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h2><p><code>sandai-react</code> re-exports everything from <code>sandai-core</code>, so you can import the client, the URL builder, the voices map, and all the types straight from this package:</p> <pre><code class="tsx"><span class="hl-3">import</span><span class="hl-1"> {</span><br/><span class="hl-1"> </span><span class="hl-4">SandaiClient</span><span class="hl-1">,</span><br/><span class="hl-1"> </span><span class="hl-4">UrlBuilder</span><span class="hl-1">,</span><br/><span class="hl-1"> </span><span class="hl-4">voices</span><span class="hl-1">,</span><br/><span class="hl-1"> </span><span class="hl-3">type</span><span class="hl-1"> </span><span class="hl-4">VoiceNames</span><span class="hl-1">,</span><br/><span class="hl-1"> </span><span class="hl-3">type</span><span class="hl-1"> </span><span class="hl-4">Environments</span><span class="hl-1">,</span><br/><span class="hl-1"> </span><span class="hl-3">type</span><span class="hl-1"> </span><span class="hl-4">AllRpcMethods</span><span class="hl-1">,</span><br/><span class="hl-1">} </span><span class="hl-3">from</span><span class="hl-1"> </span><span class="hl-2">&quot;sandai-react&quot;</span><span class="hl-1">;</span> </code><button type="button">Copy</button></pre> <a id="contributing" class="tsd-anchor"></a><h2 class="tsd-anchor-link">Contributing<a href="#contributing" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h2><p>Contributions are welcome! If you have suggestions or find issues, feel free to open an issue or a pull request.</p> <a id="license" class="tsd-anchor"></a><h2 class="tsd-anchor-link">License<a href="#license" aria-label="Permalink" class="tsd-anchor-icon"><svg viewBox="0 0 24 24" aria-hidden="true"><use href="assets/icons.svg#icon-anchor"></use></svg></a></h2><p>MIT. See the <code>LICENSE</code> file for details.</p> </div></div><div class="col-sidebar"><div class="page-menu"><div class="tsd-navigation settings"><details class="tsd-accordion"><summary class="tsd-accordion-summary"><h3><svg width="20" height="20" viewBox="0 0 24 24" fill="none" aria-hidden="true"><use href="assets/icons.svg#icon-chevronDown"></use></svg>Settings</h3></summary><div class="tsd-accordion-details"><div class="tsd-filter-visibility"><span class="settings-label">Member Visibility</span><ul id="tsd-filter-options"><li class="tsd-filter-item"><label class="tsd-filter-input"><input type="checkbox" id="tsd-filter-protected" name="protected"/><svg width="32" height="32" viewBox="0 0 32 32" aria-hidden="true"><rect class="tsd-checkbox-background" width="30" height="30" x="1" y="1" rx="6" fill="none"></rect><path class="tsd-checkbox-checkmark" d="M8.35422 16.8214L13.2143 21.75L24.6458 10.25" stroke="none" stroke-width="3.5" stroke-linejoin="round" fill="none"></path></svg><span>Protected</span></label></li><li class="tsd-filter-item"><label class="tsd-filter-input"><input type="checkbox" id="tsd-filter-inherited" name="inherited" checked/><svg width="32" height="32" viewBox="0 0 32 32" aria-hidden="true"><rect class="tsd-checkbox-background" width="30" height="30" x="1" y="1" rx="6" fill="none"></rect><path class="tsd-checkbox-checkmark" d="M8.35422 16.8214L13.2143 21.75L24.6458 10.25" stroke="none" stroke-width="3.5" stroke-linejoin="round" fill="none"></path></svg><span>Inherited</span></label></li><li class="tsd-filter-item"><label class="tsd-filter-input"><input type="checkbox" id="tsd-filter-external" name="external"/><svg width="32" height="32" viewBox="0 0 32 32" aria-hidden="true"><rect class="tsd-checkbox-background" width="30" height="30" x="1" y="1" rx="6" fill="none"></rect><path class="tsd-checkbox-checkmark" d="M8.35422 16.8214L13.2143 21.75L24.6458 10.25" stroke="none" stroke-width="3.5" stroke-linejoin="round" fill="none"></path></svg><span>External</span></label></li></ul></div><div class="tsd-theme-toggle"><label class="settings-label" for="tsd-theme">Theme</label><select id="tsd-theme"><option value="os">OS</option><option value="light">Light</option><option value="dark">Dark</option></select></div></div></details></div><details open class="tsd-accordion tsd-page-navigation"><summary class="tsd-accordion-summary"><h3><svg width="20" height="20" viewBox="0 0 24 24" fill="none" aria-hidden="true"><use href="assets/icons.svg#icon-chevronDown"></use></svg>On This Page</h3></summary><div class="tsd-accordion-details"><a href="#sandai-react"><span>sandai-<wbr/>react</span></a><ul><li><a href="#features"><span>Features</span></a></li><li><a href="#use-cases"><span>Use cases</span></a></li><li><a href="#table-of-contents"><span>Table of <wbr/>Contents</span></a></li><li><a href="#installation"><span>Installation</span></a></li><li><a href="#quick-start"><span>Quick <wbr/>Start</span></a></li><li><a href="#components"><span>Components</span></a></li><li><ul><li><a href="#ai3dcharacter"><span>AI3DCharacter</span></a></li><li><ul><li><a href="#props"><span>Props</span></a></li><li><a href="#example"><span>Example</span></a></li></ul></li><li><a href="#ai3dcharacterdocs"><span>AI3DCharacter<wbr/>Docs</span></a></li><li><ul><li><a href="#props-1"><span>Props</span></a></li><li><a href="#example-1"><span>Example</span></a></li></ul></li><li><a href="#functiontester"><span>Function<wbr/>Tester</span></a></li><li><ul><li><a href="#props-2"><span>Props</span></a></li><li><a href="#example-2"><span>Example</span></a></li></ul></li></ul></li><li><a href="#driving-the-character"><span>Driving the character</span></a></li><li><a href="#re-exports-from-sandai-core"><span>Re-<wbr/>exports from sandai-<wbr/>core</span></a></li><li><a href="#contributing"><span>Contributing</span></a></li><li><a href="#license"><span>License</span></a></li></ul></div></details></div><div class="site-menu"><nav class="tsd-navigation"><a href="modules.html">sandai-react - v2.1.8</a><ul class="tsd-small-nested-navigation" id="tsd-nav-container"><li>Loading...</li></ul></nav></div></div></div><footer><p class="tsd-generator">Generated using <a href="https://typedoc.org/" target="_blank">TypeDoc</a></p></footer><div class="overlay"></div></body></html>