llm-tokenizer
Version:
A lightweight tokenizer for OpenAI's GPT model series. Uses OpenAI's tiktoken python package
423 B
Binary
No preview is available for this file.
A lightweight tokenizer for OpenAI's GPT model series. Uses OpenAI's tiktoken python package