sandai-react/README.md

React components and utilities for the Sandai 3D AI Characters.

# sandai-react

**The React way to embed a conversational 3D AI avatar; on-device, privacy-first, one component.**

See what it's able to do in a [**custom implementation** from **DIE KAVALLERIE** 🐴](https://sustainabilitymap-mit-horst.diekavallerie.de/)

**Live playground:** [sandai.org](https://sandai.org/api_playground)

`sandai-react` is the React SDK for dropping a fully interactive **3D AI avatar** into any React or Next.js app. Mount a single `<AI3DCharacter />` component and you get a real-time **AI avatar**. A browser-based **digital human, mascot, animal, you name it**, that can speak, listen, think, and emote. Under the hood it renders the same heavily optimized runtime as [`sandai-core`](https://gitlab.com/DavidCks1/sandai-core), so you get a complete **conversational AI avatar**: on-device **text-to-speech**, **speech recognition**, an **on-device LLM**, **emotion inference**, and **lip-sync**, with no backend to run and nothing to install.

By default, **everything runs locally in the browser** via WebAssembly and WebGL/WebGPU. No API keys, no per-message costs, and no user audio ever leaves the device.

`sandai-react` re-exports all of `sandai-core` and wraps it in idiomatic React: a declarative component that builds the embed URL from props and hands you a typed `SandaiClient` in `onLoad`. It builds on the same stack: [`sandai-core`](https://gitlab.com/DavidCks1/sandai-core) → [`ai-vrm-chat`](https://gitlab.com/DavidCks1/ai-vrm-chat) → [`ai-character`](https://gitlab.com/DavidCks1/ai-character) → [`@davidcks/r3f-vrm`](https://gitlab.com/DavidCks1/r3f-vrm) (Three.js / React Three Fiber).

## Features

- **One component, zero boilerplate**: render `<AI3DCharacter />`, get a typed `SandaiClient` in `onLoad`. No iframe wiring, no `postMessage` plumbing, no 3D engine, no game engine, no local GPU server, no API gateway.
- **3D VRM avatars**: load any `.vrm` model and get a fully animated **talking avatar** with auto-blink, gaze, idle motion, and camera focus. Bring your own character, use a bundled one, or **pick from over 100,000 models** at [sandai.org](https://sandai.org/dashboard#API-Playground).
- **On-device LLM**: a **local LLM** (Gemma, Qwen, and others via transformers.js / ONNX and MediaPipe LiteRT-LM) generates replies in-browser. Or plug in your own and just stream text in.
- **Text-to-speech, 100+ voices, 30+ languages**: built-in multilingual **TTS** powered by Piper, selectable per character with the typed `voiceName` prop.
- **Real-time lip-sync**: accurate **lip-sync** driven from phonemes *or* directly from the audio waveform, with coarticulation so the mouth never looks robotic.
- **Speech recognition (ASR) + VAD**: the avatar can **listen** and reply on its own via on-device Whisper / Moonshine speech-to-text with Silero voice-activity detection.
- **Emotion inference & expression**: text is mapped to 28 GoEmotions-style **emotions** that automatically drive **facial expressions** and graded **body animations**.
- **Fully typed**: props, voices (`VoiceNames`), environments (`Environments`), and the entire `SandaiClient` RPC surface ship with TypeScript types and autocomplete.
- **WebGL & WebGPU rendering**, with an optional experimental GPU path-tracer (`raytrace`) for cinematic, ray-traced scenes.
- **Built-in dev tooling**: `AI3DCharacterDocs` renders interactive, runnable documentation for every method the character exposes — a live playground inside your own app.
- **Privacy-first, no backend**: conversation, voice, and inference happen on the user's device. Great for GDPR-sensitive products, offline-capable apps, and **keeping inference costs at zero**.

## Use cases

**AI agents** and website assistants, customer support and onboarding, **AI companions** and VTubers, interactive characters for games and education, **AI mascots** and brand mascots and virtual presenters, and any React product that benefits from a face you can talk to.

## Table of Contents

- [sandai-react](#sandai-react)
  - [Features](#features)
  - [Use cases](#use-cases)
  - [Table of Contents](#table-of-contents)
  - [Installation](#installation)
  - [Quick Start](#quick-start)
  - [Components](#components)
    - [`AI3DCharacter`](#ai3dcharacter)
    - [`AI3DCharacterDocs`](#ai3dcharacterdocs)
    - [`FunctionTester`](#functiontester)
  - [Driving the character](#driving-the-character)
  - [Re-exports from sandai-core](#re-exports-from-sandai-core)
  - [Contributing](#contributing)
  - [License](#license)

## Installation

```sh
npm install sandai-react
```

Peer dependencies (React 19):

```sh
npm install react react-dom
```

## Quick Start

Mount the component, size its container, and grab the client in `onLoad`:

```tsx
import { AI3DCharacter, SandaiClient } from "sandai-react";
import { useState } from "react";

export default function App() {
  const [client, setClient] = useState<SandaiClient>();

  return (
    <div style={{ width: "100%", height: "600px" }}>
      <AI3DCharacter
        url="https://sandai.org/chat"
        vrmUrl="https://cdn.example/ada.vrm"
        voiceName="ruri"
        environment="tokyo"
        // userId & apiKey are optional. Without them you may see a watermark.
        // Grab yours in the dashboard: https://sandai.org/dashboard
        onLoad={async (c) => {
          setClient(c);
          // Speak some text (emotion + lip-sync inferred automatically):
          const { speechEndPromise } = await c.interactionManager.say("Hi there!");
          await speechEndPromise;
          // Or have the on-device LLM reply:
          await c.interactionManager.respond("What's your name?");
        }}
      />
    </div>
  );
}
```

> The component renders an `iframe` at `width: 100%; height: 100%`, so give its parent an explicit size.
>
> Audio can't play until the user has interacted with the page (browser autoplay policy). If the character won't speak, that's almost always why — the underlying `LoadManager` tracks interaction state for you.

## Components

### `AI3DCharacter`

The one component you mount. It builds the Sandai embed URL from its props, renders the avatar in an `iframe`, constructs a [`SandaiClient`](https://gitlab.com/DavidCks1/sandai-core), and hands it to you via `onLoad` once it's ready.

```tsx
import { AI3DCharacter } from "sandai-react";
```

#### Props

| Prop | Type | Description |
|------|------|-------------|
| `url` | `string` | **Required.** Base URL of the Sandai chat interface (e.g. `https://sandai.org/chat`). |
| `onLoad` | `(client: SandaiClient) => void` | **Required.** Fired once the `SandaiClient` is initialized and ready to drive. |
| `vrmUrl` | `string` | URL of the `.vrm` model to load as the character. |
| `voiceName` | `VoiceNames` | TTS voice key (100+ multilingual Piper voices). |
| `environment` | `Environments` | Environment preset (e.g. `transparent`, `tokyo`, `sky`, `candyshop`, ...). |
| `showControls` | `boolean` | Show the debug overlay: chat input, emotion sliders, and built-in voices. Handy for previewing how the 5 VRM blendshapes combine into Sandai's 27 emotions. |
| `lowPerformanceMode` | `boolean` | Lighter render path (drops bloom/depth-of-field) for older or low-end devices. |
| `cameraType` | `"default" \| "orthographic"` | Camera projection. `default` is perspective. |
| `initialFocus` | `FocusProps` | Initial camera focus (`focusIntensity`, `cameraOffset`, `lookAtOffset`, `trackCharacterLookAt`, `focusMode`). |
| `raytrace` | `boolean` | Enable the experimental GPU path-tracer. |
| `headAnimationSmoothing` | `number` | Dampen head motion, `0..1` (0 = raw, 1 = frozen). |
| `bodyAnimationSmoothing` | `number` | Dampen body motion, `0..1`. |
| `armsAnimationSmoothing` | `number` | Dampen arm motion, `0..1`. |
| `legsAnimationSmoothing` | `number` | Dampen leg motion, `0..1`. |
| `handsAnimationSmoothing` | `number` | Dampen hand motion, `0..1`. |
| `faceAnimationSmoothing` | `number` | Dampen face/expression motion, `0..1`. Try `0.1`–`0.9` if the mouth looks choppy. |
| `decoupleOverlappingExpressionsExperimental` | `boolean` | Best-guess split of conflicting blendshapes (e.g. a model that opens its mouth when *happy*). |
| `apiKey` | `string` | Your API key (from the [dashboard](https://sandai.org/dashboard)). Optional; omit and you may see a watermark. |
| `userId` | `string` | Your user ID (from the dashboard). |
| `debugMode` | `boolean` | Skips auth + load checks. Convenient locally; **turn off in production** (it disables the interaction gate used to enable audio). |
| `style` | `React.CSSProperties` | Inline styles for the underlying `iframe`. |
| `className` | `string` | Class name for the underlying `iframe`. |

#### Example

```tsx
import { AI3DCharacter } from "sandai-react";

<AI3DCharacter
  url="https://sandai.org/chat"
  vrmUrl="https://example.com/model.vrm"
  voiceName="ruri"
  environment="tokyo"
  showControls
  faceAnimationSmoothing={0.4}
  onLoad={(client) => console.log("Sandai client ready", client)}
/>
```

---

### `AI3DCharacterDocs`

Renders **interactive, runnable documentation** for every method the loaded `SandaiClient` exposes — a live API playground you can drop straight into your own app. Pass it the `client` from `AI3DCharacter`'s `onLoad`.

#### Props

| Prop | Type | Description |
|------|------|-------------|
| `client` | `SandaiClient` | The instance from `AI3DCharacter`'s `onLoad` callback. |

#### Example

```tsx
import { AI3DCharacter, AI3DCharacterDocs } from "sandai-react";
import { useState } from "react";

const App = () => {
  const [client, setClient] = useState(null);

  return (
    <>
      <AI3DCharacter url="https://sandai.org/chat" onLoad={setClient} showControls />
      {client && <AI3DCharacterDocs client={client} />}
    </>
  );
};
```

---

### `FunctionTester`

A reusable form that dynamically generates input fields from a schema and calls a function on submit. It powers `AI3DCharacterDocs`, and you can use it standalone to build quick test harnesses around any function.

#### Props

| Prop | Type | Description |
|------|------|-------------|
| `func` | `Function` | The function to invoke; receives the form values as arguments. |
| `schema` | `SchemaType[]` | Field definitions for the function parameters. |

#### Example

```tsx
import { FunctionTester } from "sandai-react";

const myFunction = (name, age) => console.log(`Name: ${name}, Age: ${age}`);

const schema = [
  { id: "name", type: "text", label: "Name" },
  { id: "age", type: "number", label: "Age" },
];

<FunctionTester func={myFunction} schema={schema} />;
```

## Driving the character

`onLoad` hands you a `SandaiClient`, which is your control surface for the avatar. The most common calls:

```tsx
// Speak text — emotion + lip-sync are inferred automatically.
const { speechEndPromise } = await client.interactionManager.say("Hello!");
await speechEndPromise;

// Ask the on-device LLM for a reply, then speak it.
const reply = await client.interactionManager.respond("Tell me a joke about space.");

// Interrupt the current utterance.
await client.interactionManager.stop();

// Call any method on the character by typed dot-path (full autocomplete).
await client.rpcManager.call("setEmotion", "joy", 0.8);
await client.rpcManager.call("vrmManager.focusManager.focus");
```

The full `SandaiClient` API — `interactionManager`, `loadManager`, `authManager`, and the typed `rpcManager` — is documented in the [`sandai-core` README](https://gitlab.com/DavidCks1/sandai-core).

## Re-exports from sandai-core

`sandai-react` re-exports everything from `sandai-core`, so you can import the client, the URL builder, the voices map, and all the types straight from this package:

```tsx
import {
  SandaiClient,
  UrlBuilder,
  voices,
  type VoiceNames,
  type Environments,
  type AllRpcMethods,
} from "sandai-react";
```

## Contributing

Contributions are welcome! If you have suggestions or find issues, feel free to open an issue or a pull request.

## License

MIT. See the `LICENSE` file for details.