gpt-tokenizer

Version:

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

4 folders, 45 files

Filename	Content Type	Size
../
bpeRanks/	–	8.43 MB
encoding/	–	47.2 kB
encodingParams/	–	7.82 kB
model/	–	554 kB
BytePairEncodingCore.d.ts	text/typescript	2.39 kB
BytePairEncodingCore.js	text/javascript	17.1 kB
BytePairEncodingCore.js.map	application/json	13.9 kB
GptEncoding.d.ts	text/typescript	11 kB
GptEncoding.js	text/javascript	16 kB
GptEncoding.js.map	application/json	11 kB
GptEncoding.test.d.ts	text/typescript	11 B
GptEncoding.test.js	text/javascript	15.5 kB
GptEncoding.test.js.map	application/json	13.3 kB
constants.d.ts	text/typescript	105 B
constants.js	text/javascript	266 B
constants.js.map	application/json	177 B
extraApis.test.d.ts	text/typescript	11 B
extraApis.test.js	text/javascript	2.57 kB
extraApis.test.js.map	application/json	2.07 kB
main.d.ts	text/typescript	96 B
main.js	text/javascript	1.29 kB
main.js.map	application/json	179 B
mapping.d.ts	text/typescript	1 kB
mapping.js	text/javascript	1.59 kB
mapping.js.map	application/json	1.3 kB
modelParams.d.ts	text/typescript	1.15 kB
modelParams.js	text/javascript	1.27 kB
modelParams.js.map	application/json	704 B
models.d.ts	text/typescript	7.21 kB
models.js	text/javascript	27.2 kB
models.js.map	application/json	18.6 kB
otherRuntimes.test.d.ts	text/typescript	55 B
otherRuntimes.test.js	text/javascript	1.03 kB
otherRuntimes.test.js.map	application/json	903 B
resolveEncoding.d.ts	text/typescript	201 B
resolveEncoding.js	text/javascript	1.16 kB
resolveEncoding.js.map	application/json	562 B
resolveEncodingAsync.d.ts	text/typescript	215 B
resolveEncodingAsync.js	text/javascript	2.52 kB
resolveEncodingAsync.js.map	application/json	880 B
specialTokens.d.ts	text/typescript	390 B
specialTokens.js	text/javascript	594 B
specialTokens.js.map	application/json	405 B
utfUtil.d.ts	text/typescript	310 B
utfUtil.js	text/javascript	3.65 kB
utfUtil.js.map	application/json	3.05 kB
util.d.ts	text/typescript	216 B
util.js	text/javascript	695 B
util.js.map	application/json	677 B