@banuba/agora-extension
Version:
Banuba WebAR Agora extension
213 lines (146 loc) • 8.36 kB
Markdown
`@banuba/agora-extension` - Agora extension and video processor based on `@banuba/webar`.
Take a look at the [`@banuba/webar`](https://www.npmjs.com/package/@banuba/webar) npm package and [Banuba WebAR docs](https://docs.banuba.com/face-ar-sdk-v1/web/web_overview) for a deep dive.
Via `<script>` tag:
```html
<div id="video-container" style="width: 640px; height: 640px"></div>
<!-- Exposes AgoraRTC variable on window -->
<script src="https://cdn.jsdelivr.net/npm/agora-rtc-sdk-ng@4.16.1/AgoraRTC_N-production.min.js"></script>
<!-- Exposes BanubaSDK variable on window -->
<script src="https://cdn.jsdelivr.net/npm/@banuba/webar/dist/BanubaSDK.browser.js"></script>
<!-- Exposes BanubaAgoraExtension variable on window -->
<script src="https://cdn.jsdelivr.net/npm/@banuba/agora-extension/dist/index.browser.js"></script>
<script>
const banuba = new BanubaAgoraExtension.Extension({
clientToken: "xxx-xxx-xxx",
locateFile: BanubaSDK.locateFile
})
AgoraRTC.registerExtensions([banuba])
const processor = banuba.createProcessor()
await processor.addModule(
...["face_tracker", "background"].map((m) => "https://cdn.jsdelivr.net/npm/@banuba/webar/dist/modules/" + m + ".zip"),
)
const video = await AgoraRTC.createCameraVideoTrack()
video.pipe(processor).pipe(video.processorDestination)
video.play(document.querySelector("#video-container"))
</script>
```
Via `<script type="module">` tag:
```html
<div id="video-container" style="width: 640px; height: 640px"></div>
<!-- Exposes AgoraRTC variable on window -->
<script src="https://cdn.jsdelivr.net/npm/agora-rtc-sdk-ng@4.16.1/AgoraRTC_N-production.min.js"></script>
<!-- Exposes BanubaSDK variable on window -->
<script src="https://cdn.jsdelivr.net/npm/@banuba/webar/dist/BanubaSDK.browser.js"></script>
<script type="module">
import * as BanubaAgoraExtension from "https://cdn.jsdelivr.net/npm/@banuba/agora-extension/dist/index.js"
const banuba = new BanubaAgoraExtension.Extension({
clientToken: "xxx-xxx-xxx",
locateFile: BanubaSDK.locateFile
})
AgoraRTC.registerExtensions([banuba])
const processor = banuba.createProcessor()
await processor.addModule(
...["face_tracker", "background"].map(
(m) => "https://cdn.jsdelivr.net/npm/@banuba/agora-extension/dist/modules/" + m + ".zip",
),
)
const video = await AgoraRTC.createCameraVideoTrack()
video.pipe(processor).pipe(video.processorDestination)
video.play(document.querySelector("#video-container"))
</script>
```
```bash
npm i --save agora-rtc-sdk-ng @banuba/agora-extension
```
```js
import AgoraRTC from "agora-rtc-sdk-ng"
import { Extension } from "@banuba/agora-extension"
import data from "@banuba/webar/dist/BanubaSDK.data"
import wasm from "@banuba/webar/dist/BanubaSDK.wasm"
import simd from "@banuba/webar/dist/BanubaSDK.simd.wasm"
import FaceTracker from "@banuba/webar/dist/modules/face_tracker.zip"
import Background from "@banuba/webar/dist/modules/background.zip"
const banuba = new Extension({
clientToken: "xxx-xxx-xxx",
locateFile: {
"BanubaSDK.data": data,
"BanubaSDK.wasm": wasm,
"BanubaSDK.simd.wasm": simd,
},
})
AgoraRTC.registerExtensions([banuba])
const processor = banuba.createProcessor()
await processor.addModule(FaceTracker, Background)
const video = await AgoraRTC.createCameraVideoTrack()
video.pipe(processor).pipe(video.processorDestination)
// assume the page has a div element with id="video-container" and `width` and `height` set
video.play(document.querySelector("#video-container"))
```
Read the [Integration tutorials](https://docs.banuba.com/face-ar-sdk-v1/web/web_tutorials_integrations) intro for the explanation of the `locateFile` option.
Read the [Bundlers](https://docs.banuba.com/face-ar-sdk-v1/generated/typedoc/#bundlers) section of the [Integrations guides](https://docs.banuba.com/face-ar-sdk-v1/generated/typedoc/#integrations) for more receipts for different bundlers.
The `Extension` constructor accepts all the same options as the [Player.create()](https://docs.banuba.com/face-ar-sdk-v1/generated/typedoc/classes/Player.html#create) method from Banuba WebAR SDK, most noticeable options are:
- `clientToken` - Banuba Client Token, required. See [Obtaining Banuba Client token](
- `locateFile` - specifies where to look for the Banuba WebAR’s `.wasm` and `.data` files, optional. Check the [Integration tutorials](https://docs.banuba.com/face-ar-sdk-v1/web/web_tutorials_integrations) intro for explanation.
- `devicePixelRatio` - image scaling factor for HiDPI devices, optional. If not specified it has a value of `1` as a reasonable default for WebRTC AR.
Checks if the browser is capable to run the `Extension` and `Processor`:
```ts
const banuba = new Extension({
clientToken: "xxx-xxx-xxx",
locateFile: BanubaSDK.locateFile
})
if (!banuba.checkCompatibility()) alert("The browser does not support the extension :(")
```
The `Processor` instance created by the `Extension.createProcessor()` method has the following methods:
Adds AR modules (neural networks) to the processor, so effects can consume them:
```ts
// This is a syntax sugar over
// import { Module } from "@banuba/agora-extension"
// const face_tracker = new Module("/path/to/face_tracker.zip")
// await processor.addModule(face_tracker)
await processor.addModule("/path/to/face_tracker.zip")
```
See the [WebAR API Reference](https://docs.banuba.com/face-ar-sdk-v1/generated/typedoc/classes/Module.html) for the list of available AR modules and usage code examples.
Also you may want to check the docs of the [Player.addModule()](https://docs.banuba.com/face-ar-sdk-v1/generated/typedoc/classes/Player.html#addModule)from Banuba WebAR SDK as the processor’s method is a simple wrapper around the [Player.addModule()](https://docs.banuba.com/face-ar-sdk-v1/generated/typedoc/classes/Player.html#addModule).
#### `Processor.applyEffect(url | WebArEffect)`
Applies an AR effect to the processor:
```ts
// This is a syntax sugar over
// import { Effect } from "@banuba/agora-extension"
// const octopus = new Effect("/path/to/Octopus.zip")
// await processor.applyEffect(octopus)
await processor.applyEffect("/path/to/Octopus.zip")
```
You can download and use test effect from the [Demo face filters](https://docs.banuba.com/face-ar-sdk-v1/overview/demo_face_filters) page.
Also you may want to check the docs of the [Player.applyEffect()](https://docs.banuba.com/face-ar-sdk-v1/generated/typedoc/classes/Player.html#applyEffect) from Banuba WebAR SDK as the processor’s method is a simple wrapper around the [Player.applyEffect()](https://docs.banuba.com/face-ar-sdk-v1/generated/typedoc/classes/Player.html#applyEffect).
#### `Processor.clearEffect()`
Clears the applied effect if any:
```ts
await processor.applyEffect("/path/to/Octopus.zip")
// ... a few moments later ...
processor.clearEffect()
```
#### `Processor.destroy()`
Destroys the processor instance, frees up resources and allocated memory:
```ts
processor.destroy()
```
Try to not to forget to destroy the processor instance once it’s not needed to [avoid accidental memory leaks](https://docs.banuba.com/face-ar-sdk-v1/web/web_known_issues#page-running-webar-sdk-consumes-too-much-memory).
The processor instance con not be re-used once it’s destroyed.
## Obtaining Banuba Client token
[Banuba Client token](https://docs.banuba.com/face-ar-sdk-v1/overview/token_management) is required to get Banuba SDK Web AR working.
To receive a new trial client token please fill in the form on [banuba.com](https://www.banuba.com) website, or contact us via info@banuba.com.
## Resources and links
- [Banuba WebAR overview](https://docs.banuba.com/face-ar-sdk-v1/web/web_overview)
- [Banuba WebAR demo applications](https://docs.banuba.com/face-ar-sdk-v1/overview/getting_started#banuba-sdk-demo-applications)
- [Demo face filters](https://docs.banuba.com/face-ar-sdk-v1/overview/demo_face_filters)
- [Banuba WebAR JS API docs](https://docs.banuba.com/face-ar-sdk-v1/generated/typedoc/)
- [Changelog](https://docs.banuba.com/face-ar-sdk-v1/overview/releases)