@lenml/tokenizers/src/tokenizers/readme.md

a lightweight no-dependency fork of transformers.js (only tokenizers)

# tokenizers.js

this code fork from `https://github.com/huggingface/transformers`

# What changes?
- remove onnx dependencies
- remove env dependencies (esm/hf_repo_downloader)

# License
Apache-2.0