UNPKG

kitoken

Version:

Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization

Systemcluster/kitoken

23 files

	Filename	Content Type	Size
	../
	full.d.ts	text/typescript	3.88 kB
	full.js	text/javascript	190 B
	full_bg.js	text/javascript	29.1 kB
	full_bg.wasm	application/wasm	1.86 MB
	full_bg.wasm.d.ts	text/typescript	1.65 kB
	index.d.ts	text/typescript	3.88 kB
	index.js	text/javascript	194 B
	index_bg.js	text/javascript	29.1 kB
	index_bg.wasm	application/wasm	1.46 MB
	index_bg.wasm.d.ts	text/typescript	1.65 kB
	minimal.d.ts	text/typescript	2.78 kB
	minimal.js	text/javascript	202 B
	minimal_bg.js	text/javascript	22.1 kB
	minimal_bg.wasm	application/wasm	991 kB
	minimal_bg.wasm.d.ts	text/typescript	1.06 kB
	node.cjs	text/javascript	30.7 kB
	node.d.ts	text/typescript	3.88 kB
	node_bg.wasm	application/wasm	1.86 MB
	node_bg.wasm.d.ts	text/typescript	1.65 kB
	web.d.ts	text/typescript	6.47 kB
	web.js	text/javascript	33.9 kB
	web_bg.wasm	application/wasm	1.46 MB
	web_bg.wasm.d.ts	text/typescript	1.65 kB