gpt-tokenizer

Version:

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

5 folders, 58 files

Filename	Content Type	Size
../
bpeRanks/	–	8.43 MB
codegen/	–	27.9 kB
encoding/	–	32.1 kB
encodingParams/	–	7.48 kB
model/	–	645 kB
BytePairEncodingCore.d.ts	text/typescript	2.39 kB
BytePairEncodingCore.js	text/javascript	16.9 kB
BytePairEncodingCore.js.map	application/json	14 kB
GptEncoding.d.ts	text/typescript	10.2 kB
GptEncoding.js	text/javascript	16.2 kB
GptEncoding.js.map	application/json	11.7 kB
GptEncoding.test.d.ts	text/typescript	11 B
GptEncoding.test.js	text/javascript	14.1 kB
GptEncoding.test.js.map	application/json	12.7 kB
constants.d.ts	text/typescript	105 B
constants.js	text/javascript	127 B
constants.js.map	application/json	194 B
extraApis.test.d.ts	text/typescript	11 B
extraApis.test.js	text/javascript	2.07 kB
extraApis.test.js.map	application/json	1.96 kB
main.d.ts	text/typescript	94 B
main.js	text/javascript	202 B
main.js.map	application/json	177 B
mapping.d.ts	text/typescript	1.07 kB
mapping.js	text/javascript	1.24 kB
mapping.js.map	application/json	1.23 kB
modelParams.d.ts	text/typescript	1.23 kB
modelParams.js	text/javascript	1.01 kB
modelParams.js.map	application/json	812 B
modelTypes.d.ts	text/typescript	2.72 kB
modelTypes.js	text/javascript	49 B
modelTypes.js.map	application/json	112 B
models.d.ts	text/typescript	25 kB
models.gen.d.ts	text/typescript	43.6 kB
models.gen.js	text/javascript	102 kB
models.gen.js.map	application/json	63.4 kB
models.js	text/javascript	33.9 kB
models.js.map	application/json	28.3 kB
modelsChatEnabled.gen.d.ts	text/typescript	1.36 kB
modelsChatEnabled.gen.js	text/javascript	1.39 kB
modelsChatEnabled.gen.js.map	application/json	873 B
modelsMap.d.ts	text/typescript	258 B
modelsMap.js	text/javascript	2.24 kB
modelsMap.js.map	application/json	1.21 kB
otherRuntimes.test.d.ts	text/typescript	55 B
otherRuntimes.test.js	text/javascript	833 B
otherRuntimes.test.js.map	application/json	976 B
package.json	application/json	44 B
resolveEncoding.d.ts	text/typescript	201 B
resolveEncoding.js	text/javascript	676 B
resolveEncoding.js.map	application/json	620 B
resolveEncodingAsync.d.ts	text/typescript	215 B
resolveEncodingAsync.js	text/javascript	819 B
resolveEncodingAsync.js.map	application/json	864 B
specialTokens.d.ts	text/typescript	390 B
specialTokens.js	text/javascript	397 B
specialTokens.js.map	application/json	482 B
utfUtil.d.ts	text/typescript	310 B
utfUtil.js	text/javascript	3.37 kB
utfUtil.js.map	application/json	3.01 kB
util.d.ts	text/typescript	216 B
util.js	text/javascript	500 B
util.js.map	application/json	653 B