@vsllabs/webgl-react/README.md

VSL-Labs Webgl implementation for translating text into 3D sign language

# @vsllabs/webgl-react


## Installation:

To install the package, run:

Using npm:
```bash
$ npm install @vsllabs/webgl-react
```

Using yarn:
```bash
$ yarn add @vsllabs/webgl-react
```


## Table of Contents

- [Usage Example (Normal mode)](#usage-example-normal-mode)
- [Usage Example (Streaming mode)](#usage-example-Streaming-mode)
- [Documentation](#documentation)
- [Required Parameters](#required-parameters)
- [Returned Values](#returned-values)
- [Example Workflow](#example-workflow)


## Usage Example (Normal mode):

Below is an example of how to use the useVslWebGL hook within a React component:


```js
import { useState } from 'react'
import { useVslWebGL } from '@vsllabs/webgl-react';

const app = () => {
    // example text input state
    const [inputText, setInputText] = useState('')

    // invoke the useVslWebGL hook with your personal API_KEY and build URLs
    const { VSLWebGl, unityProvider, translateTextToASL, isUnityLoaded, isTranslating, replay } = useVslWebGL({
        API_KEY: 'Your API Key here',
        streaming_mode: false,    // Optional (false by default)
        // *We will provide these URLs for you
        loaderUrl: 'Unity build Url',
        dataUrl: 'Unity build Url',
        frameworkUrl: 'Unity build Url',
        codeUrl: 'Unity build Url',
     })

  return (
    <div
      style={{
        width: "400px",
        height: "700px",
        position: "relative",
        margin: "auto",
      }}
    >
      {!isUnityLoaded && (
        <div
          style={{
            position: "absolute",
            width: "100%",
            height: "100%",
            top: 0,
            left: 0,
            background: "#999",
            display: "grid",
            placeItems: "center",
          }}
        >
          {/* Your Loading spinner here */}
          Loading...
        </div>
      )}

      {/* The WebGL component for rendering, can be styled as needed */}
      <VSLWebGl
        id="unityWebGl"
        style={{ width: "100%", height: "100%" }}
        unityProvider={unityProvider}
      />

      {/* Example input for the text state */}
      <input
        type="text"
        placeholder="Enter Text..."
        value={inputText}
        onChange={(ev) => setInputText(ev.target.value)}
        style={{
          border: "1px solid black",
          borderRadius: "8px",
          minHeight: "30px",
          width: "100%",
          marginBottom: "16px",
          padding: "8px",
        }}
      />

      {/* Translate button, triggers translation when Unity is loaded and input is provided */}
      <button
        style={{
          border: "1px solid black",
          borderRadius: "4px",
          padding: "8px 16px",
          marginRight: "16px",
          backgroundColor: "green",
          color: "white",
        }}
        type="button"
        onClick={() => translateTextToASL(inputText)}
      >
        {isTranslating ? "Loading..." : "Translate"}
      </button>

      {/* Replay button, replays the last translation */}
      <button
        style={{
          border: "1px solid black",
          borderRadius: "4px",
          padding: "8px 16px",
        }}
        type="button"
        onClick={replay}
      >
        Replay
      </button>
    </div>
  );
}
```

## Usage Example (Streaming mode):

Below is an example of the streaming mode with basic UI using the browser speech recognition for illustration:


```js
import { useState, useEffect } from "react";
import { useVslWebGL } from "@vsllabs/webgl-react";
import "./App.css"; // Importing the styling CSS file (file content is below)

const App = () => {
  const [inputText, setInputText] = useState("");
  const [isStreaming, setIsStreaming] = useState(false);

  const {
    VSLWebGl,
    unityProvider,
    translateTextToASL,
    isUnityLoaded,
    isTranslating,
    replay,
  } = useVslWebGL({
        API_KEY: 'Your API Key here',
        streaming_mode: isStreaming,
        // *We will provide these URLs for you
        loaderUrl: 'Unity build Url',
        dataUrl: 'Unity build Url',
        frameworkUrl: 'Unity build Url',
        codeUrl: 'Unity build Url',
     });

  useEffect(() => {
    if (isStreaming) {
      const SpeechRecognition =
        window.SpeechRecognition || window.webkitSpeechRecognition;

      if (!SpeechRecognition) {
        alert("Speech Recognition is not supported in this browser.");
        return;
      }

      const recognition = new SpeechRecognition();
      recognition.continuous = true;
      recognition.interimResults = false;
      recognition.lang = "en-US";

      recognition.onresult = (event) => {
        const lastResult = event.results[event.results.length - 1];
        if (lastResult.isFinal) {
          const spokenText = lastResult[0].transcript.trim();
          setInputText(spokenText);
          translateTextToASL(spokenText);
        }
      };

      recognition.onerror = (event) => {
        console.error(
          "Speech recognition error:",
          event.error || event.message || event
        );
      };

      recognition.start();

      return () => recognition.stop();
    }
  }, [isStreaming]);

  return (
    <div id="appContainer">
      {!isUnityLoaded && (
        <div id="loadingOverlay">
          <div className="spinner" />
        </div>
      )}

      <div id="webglContainer">
        <VSLWebGl
          id="unityWebGl"
          style={{ width: "100%", height: "100%" }}
          unityProvider={unityProvider}
        />
      </div>

      <input
        id="textInput"
        type="text"
        value={inputText}
        onChange={(e) => setInputText(e.target.value)}
        placeholder="Enter text to translate"
      />

      <div id="buttonGroup">
        <button
          id="translateBtn"
          type="button"
          onClick={() => translateTextToASL(inputText)}
          disabled={!isUnityLoaded || isTranslating}
          className={isTranslating ? "btn translate disabled" : "btn translate"}
        >
          {isTranslating ? "Loading..." : "Translate"}
        </button>

        <button
          id="replayBtn"
          type="button"
          onClick={replay}
          className="btn replay"
          className={isStreaming ? "btn replay disabled" : "btn replay"}
          disabled={isStreaming}
        >
          Replay
        </button>

        <button
          id="streamBtn"
          type="button"
          onClick={() => setIsStreaming((prev) => !prev)}
          className={`btn stream ${isStreaming ? "stop" : "start"}`}
        >
          {isStreaming ? "Stop Streaming" : "Start Streaming"}
        </button>
      </div>
    </div>
  );
};

export default App;
```

### Streaming mode app.css file:


```css
#appContainer {
  width: 800px;
  height: 850px;
  margin: 0 auto;
  position: relative;
  display: flex;
  flex-direction: column;
  align-items: center;
  gap: 1rem;
  padding: 1rem;
  background-color: white;
  border-radius: 1rem;
  box-shadow: 0 10px 15px rgba(0, 0, 0, 0.1);
}

#loadingOverlay {
  position: absolute;
  top: 0;
  left: 0;
  width: 100%;
  height: 100%;
  background-color: #d1d5db;
  display: flex;
  align-items: center;
  justify-content: center;
  border-radius: 1rem;
  z-index: 10;
}

.spinner {
  animation: spin 1s linear infinite;
  border-top: 4px solid #3b82f6;
  border-radius: 9999px;
  width: 3rem;
  height: 3rem;
  border-right: 4px solid transparent;
}

@keyframes spin {
  to {
    transform: rotate(360deg);
  }
}

#webglContainer {
  width: 600px;
  height: 600px;
  position: relative;
  border-radius: 1rem;
  overflow: hidden;
}

#textInput {
  width: 100%;
  padding: 0.5rem 1rem;
  border: 1px solid #d1d5db;
  border-radius: 0.375rem;
  outline: none;
  transition: box-shadow 0.2s;
}

#textInput:focus {
  box-shadow: 0 0 0 2px #3b82f6;
}

#buttonGroup {
  display: flex;
  gap: 0.5rem;
  width: 100%;
}

.btn {
  flex: 1;
  padding: 0.5rem 1rem;
  border-radius: 0.375rem;
  color: white;
  transition: background-color 0.2s;
  cursor: pointer;
  border: none;
}

.btn.translate {
  background-color: #2563eb;
}

.btn.translate:hover {
  background-color: #1d4ed8;
}

.btn.translate.disabled {
  background-color: #93c5fd;
  cursor: not-allowed;
}

.btn.replay {
  background-color: #6b7280;
}

.btn.replay.disabled {
  background-color: #5c616b;
  cursor: not-allowed;
}

.btn.replay:hover {
  background-color: #4b5563;
}

.btn.stream.start {
  background-color: #16a34a;
}

.btn.stream.start:hover {
  background-color: #15803d;
}

.btn.stream.stop {
  background-color: #dc2626;
}

.btn.stream.stop:hover {
  background-color: #b91c1c;
}

```

## Documentation

The ***useVslWebGL*** hook provides the necessary setup and functionality for integrating the VSL WebGL component within a React application. It returns an object with various properties and functions for rendering, controlling, and interacting with the WebGL component.


### Required Parameters
 * **API_KEY**: Your unique API key for accessing the VSL WebGL services.
 * **loaderUrl, dataUrl, frameworkUrl, codeUrl**: URLs provided by VSL for accessing the Unity WebGL build. Each URL is necessary for loading the Unity environment properly.

### Optional Parameters
 * **streaming_mode**: Toggle streaming mode. (boolean)


### Returned Values

The following values and functions are returned by useVslWebGL:

| Value         | Explanation                                    |
| :------------ | :--------------------------------------------- |
| VSLWebGl | <ul><li>A JSX component for rendering the Unity WebGL. Can be styled and controlled within a parent component or container.</li><li>Example: <br><pre lang="javascript"> <VSLWebGl style={{ ... }} unityProvider={unityProvider} /> </pre></li></ul>  |
| unityProvider | <ul><li>Required prop for the ***VSLWebGl*** component, provides the Unity instance.</li><li>Pass this to the ***unityProvider*** prop of *VSLWebGl* to initialize the Unity environment.</li></ul>  |
| translateTextToASL | <ul><li>Function to trigger text translation within the Unity WebGL.</li><li>**Arguments:** Accepts a single argument (the text to translate).</li><li>**Example:** <pre lang="javascript"> translateTextToASL("Hello, world!") </pre></li></ul>  |
| isUnityLoaded | <ul><li>Indicates whether the Unity WebGL component has fully loaded. Useful for checking readiness to show loaders and before triggering translation.</li><li>**Example:** Disabling the translate button until Unity is ready.</li></ul>  |
| isTranslating | <ul><li>Represents the loading state during the translation process. Helpful for displaying loading indicators.</li><li>**Example:** <pre lang="javascript"> {isTranslating ? 'Translating...' : 'Translate'} </pre></li></ul>  |
| replay | <ul><li>Function to replay the last translated text within the Unity WebGL.</li><li>**Arguments:** No arguments required.</li></ul>  |
| changeBgColor | <ul><li>Changes the background color of the WebGL component.</li><li>**Arguments:** Accepts a single argument, a string representing a hex color value (e.g., #FFFFFF for white).</li><li>**Example:** <pre lang="javascript"> changeBgColor('#FF5733') </pre> to set the background color to a shade of orange.</li></ul>  |
| setAnimationSpeed | <ul><li>Controls the speed of animations within the WebGL environment.</li><li>**Arguments:** Accepts a single string argument, which can be one of four options: "0" (for pause), "0.5", "1" (default), or "1.5".</li><li>**Example:** <pre lang="javascript"> setAnimationSpeed("1.5") </pre> to set the animation speed to 1.5x.</li></ul>  |
| toggleCameraRotation | <ul><li>Toggles the rotation of the camera in the WebGL environment.</li><li>**Arguments:** Accepts a single boolean argument to enable (true) or disable (false > default) camera rotation.</li><li>**Example:** <pre lang="javascript"> toggleCameraRotation(true) </pre> to enable camera rotation.</li></ul>  |
| error | <ul><li>If any errors occur during loading or translation, this string provides an error message explaining the issue.</li><li>**Example:** Display error in your UI if it’s not an empty string.</li><li>**Note:** Errors are also logged in the console</li></ul>  |
| streaming_mode | <ul><li>Set to true to use streaming mode. </li><li>**Example:** When streaming mode is on, text provided to ***translateTextToASL*** function will be added to a queue where the animations will play in sequential manner. </li><li>**Note:** When set to false and a new text is provided to ***translateTextToASL*** while an animation is playing, will override it and start animating the new sentence. </li><li>**Note:** When streaming mode is on the ***reply*** function cannot be used, invoking it will do nothing</li></ul>  |


### Example Workflow

 1. Initialize the Hook: Call useVslWebGL with the required parameters to initialize the WebGL component.
 2. Render the Component: Use <VSLWebGl /> in your component, styled to fit your layout.
 3. Translate Text: Use the translateTextToASL function to translate input text when Unity is loaded (isUnityLoaded).
 4. Replay Last Translation: Use the replay function to repeat the last translation as needed.
 5. Handle Errors: Check the error value to catch and display any issues that occur during loading or translation.


 ## License

[MIT](LICENSE)