UNPKG

gpt-tokenizer

Version:

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

github.com/niieani/gpt-tokenizer

niieani/gpt-tokenizer

4 lines (3 loc) • 214 B
1
2
3
4
import type { RawBytePairRanks } from '../BytePairEncodingCore.js'; import { type EncodingParams } from '../modelParams.js'; export declare function P50KEdit(bytePairRankDecoder: RawBytePairRanks): EncodingParams;