gpt-tokenizer

Version:

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

3 lines (2 loc) • 112 B

TypeScript

export declare const R50K_TOKEN_SPLIT_REGEX: RegExp; export declare const CL_AND_O_TOKEN_SPLIT_PATTERN: RegExp;