gpt-tokenizer

Version:

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

5 folders, 46 files

Filename	Content Type	Size
../
bpeRanks/	–	8.43 MB
codegen/	–	10.9 kB
encoding/	–	39.5 kB
encodingParams/	–	7.48 kB
model/	–	434 kB
BytePairEncodingCore.d.ts	text/typescript	2.39 kB
BytePairEncodingCore.js	text/javascript	16.9 kB
BytePairEncodingCore.js.map	application/json	14 kB
GptEncoding.d.ts	text/typescript	11 kB
GptEncoding.js	text/javascript	15.5 kB
GptEncoding.js.map	application/json	11.3 kB
GptEncoding.test.d.ts	text/typescript	11 B
GptEncoding.test.js	text/javascript	13.8 kB
GptEncoding.test.js.map	application/json	13.1 kB
constants.d.ts	text/typescript	105 B
constants.js	text/javascript	127 B
constants.js.map	application/json	194 B
extraApis.test.d.ts	text/typescript	11 B
extraApis.test.js	text/javascript	2.22 kB
extraApis.test.js.map	application/json	2.09 kB
main.d.ts	text/typescript	96 B
main.js	text/javascript	204 B
main.js.map	application/json	177 B
mapping.d.ts	text/typescript	1 kB
mapping.js	text/javascript	1.25 kB
mapping.js.map	application/json	1.45 kB
modelParams.d.ts	text/typescript	1.15 kB
modelParams.js	text/javascript	1.01 kB
modelParams.js.map	application/json	812 B
models.d.ts	text/typescript	7.21 kB
models.js	text/javascript	27 kB
models.js.map	application/json	18.6 kB
otherRuntimes.test.d.ts	text/typescript	55 B
otherRuntimes.test.js	text/javascript	833 B
otherRuntimes.test.js.map	application/json	976 B
package.json	application/json	44 B
resolveEncoding.d.ts	text/typescript	201 B
resolveEncoding.js	text/javascript	676 B
resolveEncoding.js.map	application/json	620 B
resolveEncodingAsync.d.ts	text/typescript	215 B
resolveEncodingAsync.js	text/javascript	819 B
resolveEncodingAsync.js.map	application/json	864 B
specialTokens.d.ts	text/typescript	390 B
specialTokens.js	text/javascript	397 B
specialTokens.js.map	application/json	482 B
utfUtil.d.ts	text/typescript	310 B
utfUtil.js	text/javascript	3.37 kB
utfUtil.js.map	application/json	3.01 kB
util.d.ts	text/typescript	216 B
util.js	text/javascript	500 B
util.js.map	application/json	653 B