UNPKG
@lenml/tokenizer-claude
Version:
latest (3.7.2)
3.7.2
3.4.2
3.4.1
3.4.0
3.0.1
1.1.2
1.1.1
1.0.10
1.0.9
1.0.8
1.0.7
1.0.4
1.0.3
1.0.1
claude tokenizer for NodeJS/Browser
github.com/lenML/tokenizers
lenML/tokenizers
@lenml/tokenizer-claude
/
models
/
tokenizer_config.json
9 lines
•
213 B
JSON
View Raw
1
2
3
4
5
6
7
8
9
{
"add_prefix_space"
:
false
,
"bos_token"
:
"<EOT>"
,
"clean_up_tokenization_spaces"
:
true
,
"eos_token"
:
"<EOT>"
,
"model_max_length"
:
200000
,
"tokenizer_class"
:
"ClaudeTokenizer"
,
"unk_token"
:
"<EOT>"
}