UNPKG

gemini-token-estimator

Version:

Estimate the number of tokens for Gemini models

github.com/smartprocure/gemini-token-estimator

smartprocure/gemini-token-estimator

25 lines (13 loc) • 345 B
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
# 0.6.0 - Exclude private use area from non-Lation regex. # 0.5.0 - Handle this character properly:  # 0.4.0 - Handle this character properly:  # 0.3.0 - Properly handle French, German, and Spanish characters. - Improved logic for non-Latin characters. # 0.2.0 - Properly handle non-Latin characters. # 0.1.0 - Initial release.