UNPKG

gpt-tokenizer

Version:

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

github.com/niieani/gpt-tokenizer

niieani/gpt-tokenizer

6 folders, 20 files

	Filename	Content Type	Size
	../
	__snapshots__/	–	118 kB
	bpeRanks/	–	3.92 MB
	codegen/	–	14.3 kB
	encoding/	–	4.02 kB
	encodingParams/	–	3.77 kB
	model/	–	91.3 kB
	BytePairEncodingCore.ts	text/typescript	16.5 kB
	GptEncoding.test.ts	text/typescript	13.4 kB
	GptEncoding.ts	text/typescript	17.3 kB
	constants.ts	text/typescript	88 B
	extraApis.test.ts	text/typescript	1.95 kB
	main.ts	text/typescript	168 B
	mapping.ts	text/typescript	1.62 kB
	modelParams.ts	text/typescript	2.04 kB
	modelTypes.ts	text/typescript	2.68 kB
	models.gen.ts	text/typescript	90.1 kB
	models.ts	text/typescript	34.6 kB
	modelsChatEnabled.gen.ts	text/typescript	1.29 kB
	modelsMap.ts	text/typescript	2.38 kB
	otherRuntimes.test.ts	text/typescript	746 B
	package.json	application/json	72 B
	resolveEncoding.ts	text/typescript	709 B
	resolveEncodingAsync.ts	text/typescript	972 B
	specialTokens.ts	text/typescript	348 B
	utfUtil.ts	text/typescript	3.01 kB
	util.ts	text/typescript	501 B