UNPKG

llm-tokenizer

Version:

A lightweight tokenizer for OpenAI's GPT model series. Uses OpenAI's tiktoken python package

github.com/KevinKWZheng/OpenAI-Tokenizer

KevinKWZheng/OpenAI-Tokenizer

4 files

	Filename	Content Type	Size
	../
	countToken.py	application/octet-stream	434 B
	decodeText.py	application/octet-stream	423 B
	encodeText.py	application/octet-stream	395 B
	getEncoding.py	application/octet-stream	289 B