UNPKG

kitoken

Version:

Fast and versatile tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization

Systemcluster/kitoken

24 files

	Filename	Content Type	Size
	../
	.gitignore	text/plain	1 B
	full.d.ts	text/typescript	2.82 kB
	full.js	text/javascript	138 B
	full_bg.js	text/javascript	27.5 kB
	full_bg.wasm	application/wasm	1.67 MB
	full_bg.wasm.d.ts	text/typescript	1.61 kB
	index.d.ts	text/typescript	2.82 kB
	index.js	text/javascript	141 B
	index_bg.js	text/javascript	27.5 kB
	index_bg.wasm	application/wasm	1.43 MB
	index_bg.wasm.d.ts	text/typescript	1.61 kB
	minimal.d.ts	text/typescript	1.71 kB
	minimal.js	text/javascript	147 B
	minimal_bg.js	text/javascript	19.2 kB
	minimal_bg.wasm	application/wasm	943 kB
	minimal_bg.wasm.d.ts	text/typescript	960 B
	node.cjs	text/javascript	28.1 kB
	node.d.ts	text/typescript	2.82 kB
	node_bg.wasm	application/wasm	1.67 MB
	node_bg.wasm.d.ts	text/typescript	1.61 kB
	web.d.ts	text/typescript	5.31 kB
	web.js	text/javascript	31.8 kB
	web_bg.wasm	application/wasm	1.43 MB
	web_bg.wasm.d.ts	text/typescript	1.61 kB