UNPKG
@lenml/tokenizer-gpt2
Version:
latest (3.7.2)
3.7.2
3.4.2
3.4.1
3.4.0
3.0.1
1.1.2
1.1.1
1.0.10
1.0.9
1.0.8
1.0.4
1.0.3
1.0.1
gpt2 tokenizer for NodeJS/Browser
github.com/lenML/tokenizers
lenML/tokenizers
@lenml/tokenizer-gpt2
/
models
/
tokenizer_config.json
10 lines
(9 loc)
•
234 B
JSON
View Raw
1
2
3
4
5
6
7
8
9
10
{
"add_prefix_space"
:
false
,
"bos_token"
:
"<|endoftext|>"
,
"clean_up_tokenization_spaces"
:
true
,
"eos_token"
:
"<|endoftext|>"
,
"model_max_length"
:
1024
,
"tokenizer_class"
:
"GPT2Tokenizer"
,
"unk_token"
:
"<|endoftext|>"
}