UNPKG

gpt-tokenizer

Version:

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

github.com/niieani/gpt-tokenizer

niieani/gpt-tokenizer

18 files

	Filename	Content Type	Size
	../
	cl100k_base.d.ts	text/typescript	216 B
	cl100k_base.js	text/javascript	907 B
	cl100k_base.js.map	application/json	603 B
	constants.d.ts	text/typescript	112 B
	constants.js	text/javascript	467 B
	constants.js.map	application/json	201 B
	o200k_base.d.ts	text/typescript	215 B
	o200k_base.js	text/javascript	903 B
	o200k_base.js.map	application/json	601 B
	p50k_base.d.ts	text/typescript	214 B
	p50k_base.js	text/javascript	529 B
	p50k_base.js.map	application/json	360 B
	p50k_edit.d.ts	text/typescript	214 B
	p50k_edit.js	text/javascript	701 B
	p50k_edit.js.map	application/json	470 B
	r50k_base.d.ts	text/typescript	214 B
	r50k_base.js	text/javascript	529 B
	r50k_base.js.map	application/json	360 B