@chonkiejs/token
Version:
HuggingFace tokenizer support for Chonkie - extends @chonkiejs/core with real tokenization
| Filename | Content Type | Size | |
|---|---|---|---|
| 1.06 kB | |||
| 3.38 kB | |||
| 1.27 kB |
HuggingFace tokenizer support for Chonkie - extends @chonkiejs/core with real tokenization
| Filename | Content Type | Size | |
|---|---|---|---|
| 1.06 kB | |||
| 3.38 kB | |||
| 1.27 kB |