UNPKG

@agnai/web-tokenizers

Version:

| [NPM Package](https://www.npmjs.com/package/@mlc-ai/web-tokenizers) | [WebLLM](https://github.com/mlc-ai/web-llm) |

github.com/mlc-ai/tokenizers-cpp/tree/main/web

mlc-ai/tokenizers-cpp

23 lines (17 loc) • 581 B
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# WebTokenizers | [NPM Package](https://www.npmjs.com/package/@mlc-ai/web-tokenizers) | [WebLLM](https://github.com/mlc-ai/web-llm) | WebTokenizers is a javascript binding library that can be universally deployed. It wraps and binds the [HuggingFace tokenizers library](https://github.com/huggingface/tokenizers) and [sentencepiece](https://github.com/google/sentencepiece) and provides a minimum common interface in havascript. ## Build from Srouce ```bash source /path/to/emsdk_env.sh npm install npm run build ``` To try out the test webpage ```bash cd tests npm start ```