llm-tokenizer

Version:

A lightweight tokenizer for OpenAI's GPT model series. Uses OpenAI's tiktoken python package

3 lines (2 loc) • 66 B

Plain Text

# Auto detect text files and perform LF normalization * text=auto