UNPKG

gpt-tokenizer

Version:

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

github.com/niieani/gpt-tokenizer

niieani/gpt-tokenizer

12 files

	Filename	Content Type	Size
	../
	generateByModel.d.ts	text/typescript	11 B
	generateByModel.js	text/javascript	1.66 kB
	generateByModel.js.map	application/json	1.52 kB
	generateChatEnabled.d.ts	text/typescript	11 B
	generateChatEnabled.js	text/javascript	820 B
	generateChatEnabled.js.map	application/json	931 B
	generateJsBpe.d.ts	text/typescript	11 B
	generateJsBpe.js	text/javascript	4.06 kB
	generateJsBpe.js.map	application/json	4.17 kB
	modelScape.d.ts	text/typescript	11 B
	modelScape.js	text/javascript	8.24 kB
	modelScape.js.map	application/json	6.44 kB