gpt-tokenizer

Version:

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

4 folders, 57 files

Filename	Content Type	Size
../
bpeRanks/	–	8.43 MB
encoding/	–	39.8 kB
encodingParams/	–	7.82 kB
model/	–	807 kB
BytePairEncodingCore.d.ts	text/typescript	2.39 kB
BytePairEncodingCore.js	text/javascript	17.1 kB
BytePairEncodingCore.js.map	application/json	13.9 kB
GptEncoding.d.ts	text/typescript	10.2 kB
GptEncoding.js	text/javascript	16.6 kB
GptEncoding.js.map	application/json	11.4 kB
GptEncoding.test.d.ts	text/typescript	11 B
GptEncoding.test.js	text/javascript	17 kB
GptEncoding.test.js.map	application/json	12.7 kB
constants.d.ts	text/typescript	105 B
constants.js	text/javascript	266 B
constants.js.map	application/json	177 B
extraApis.test.d.ts	text/typescript	11 B
extraApis.test.js	text/javascript	2.76 kB
extraApis.test.js.map	application/json	2 kB
main.d.ts	text/typescript	94 B
main.js	text/javascript	1.28 kB
main.js.map	application/json	179 B
mapping.d.ts	text/typescript	1.07 kB
mapping.js	text/javascript	2.98 kB
mapping.js.map	application/json	1.1 kB
modelParams.d.ts	text/typescript	1.23 kB
modelParams.js	text/javascript	1.27 kB
modelParams.js.map	application/json	704 B
modelTypes.d.ts	text/typescript	2.72 kB
modelTypes.js	text/javascript	115 B
modelTypes.js.map	application/json	112 B
models.d.ts	text/typescript	25 kB
models.gen.d.ts	text/typescript	43.6 kB
models.gen.js	text/javascript	104 kB
models.gen.js.map	application/json	60.5 kB
models.js	text/javascript	35.9 kB
models.js.map	application/json	26.9 kB
modelsChatEnabled.gen.d.ts	text/typescript	1.36 kB
modelsChatEnabled.gen.js	text/javascript	1.5 kB
modelsChatEnabled.gen.js.map	application/json	866 B
modelsMap.d.ts	text/typescript	258 B
modelsMap.js	text/javascript	2.4 kB
modelsMap.js.map	application/json	1.16 kB
otherRuntimes.test.d.ts	text/typescript	55 B
otherRuntimes.test.js	text/javascript	1.03 kB
otherRuntimes.test.js.map	application/json	903 B
resolveEncoding.d.ts	text/typescript	201 B
resolveEncoding.js	text/javascript	1.16 kB
resolveEncoding.js.map	application/json	562 B
resolveEncodingAsync.d.ts	text/typescript	215 B
resolveEncodingAsync.js	text/javascript	2.52 kB
resolveEncodingAsync.js.map	application/json	880 B
specialTokens.d.ts	text/typescript	390 B
specialTokens.js	text/javascript	594 B
specialTokens.js.map	application/json	405 B
utfUtil.d.ts	text/typescript	310 B
utfUtil.js	text/javascript	3.65 kB
utfUtil.js.map	application/json	3.05 kB
util.d.ts	text/typescript	216 B
util.js	text/javascript	695 B
util.js.map	application/json	677 B