llm-tokenizer
Version:
A lightweight tokenizer for OpenAI's GPT model series. Uses OpenAI's tiktoken python package
| Filename | Content Type | Size | |
|---|---|---|---|
| ../ | |||
| 2.34 kB | |||
| 310 B | |||
| 1.04 kB | |||
| 3.64 kB | |||
A lightweight tokenizer for OpenAI's GPT model series. Uses OpenAI's tiktoken python package
| Filename | Content Type | Size | |
|---|---|---|---|
| ../ | |||
| 2.34 kB | |||
| 310 B | |||
| 1.04 kB | |||
| 3.64 kB | |||