UNPKG

gpt-tokenizer

Version:

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

github.com/niieani/gpt-tokenizer

niieani/gpt-tokenizer

6 files

	Filename	Content Type	Size
	../
	generateByModel.d.ts	text/typescript	11 B
	generateByModel.js	text/javascript	1.34 kB
	generateByModel.js.map	application/json	1.28 kB
	generateJsBpe.d.ts	text/typescript	11 B
	generateJsBpe.js	text/javascript	4.06 kB
	generateJsBpe.js.map	application/json	4.17 kB