UNPKG

gpt-tokenizer

Version:

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

github.com/niieani/gpt-tokenizer

niieani/gpt-tokenizer

6 lines • 266 B
1
2
3
4
5
6
"use strict"; Object.defineProperty(exports, "__esModule", { value: true }); exports.DEFAULT_MERGE_CACHE_SIZE = exports.ALL_SPECIAL_TOKENS = void 0; exports.ALL_SPECIAL_TOKENS = 'all'; exports.DEFAULT_MERGE_CACHE_SIZE = 100_000; //# sourceMappingURL=constants.js.map