@anush008/tokenizers

Version:

Multi-arch builds of HuggingFace tokenizers

1 folder, 11 files

Filename	Content Type	Size
../
tasks/	–	4.94 kB
arc_rwlock_serde.rs	application/rls-services+xml	506 B
decoders.rs	application/rls-services+xml	4.26 kB
encoding.rs	application/rls-services+xml	8.14 kB
lib.rs	application/rls-services+xml	283 B
models.rs	application/rls-services+xml	8.84 kB
normalizers.rs	application/rls-services+xml	4.62 kB
pre_tokenizers.rs	application/rls-services+xml	7.19 kB
processors.rs	application/rls-services+xml	3.28 kB
tokenizer.rs	application/rls-services+xml	13.2 kB
trainers.rs	application/rls-services+xml	1.37 kB
utils.rs	application/rls-services+xml	1.21 kB