UNPKG

gpt-tokenizer

Version:

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

github.com/niieani/gpt-tokenizer

niieani/gpt-tokenizer

6 lines (5 loc) • 258 B
1
2
3
4
5
6
export declare const p50k_base: readonly string[]; export declare const r50k_base: readonly string[]; export declare const p50k_edit: readonly string[]; export declare const cl100k_base: readonly string[]; export declare const o200k_base: readonly string[];