tokenx/README.md

Fast token estimation with 95%+ average accuracy, 2kB minified and gzipped

# tokenx

Fast and lightweight token count estimation without requiring a full tokenizer.

Estimates are calibrated against OpenAI's `o200k_base` encoding – the tokenizer of all current GPT models. Counts for other LLM families differ somewhat; the `defaultCharsPerToken` and `languageConfigs` options let you tune the heuristics for your model. For precise counts, use a full tokenizer like [`gpt-tokenizer`](https://github.com/niieani/gpt-tokenizer).

## Features

- 🎯 **95%+ average accuracy**, and no single sample below 90%
- 📦 **Just 2kB** minified and gzipped, with zero dependencies
- 🖥️ **[Bundled CLI](#cli)** – count, slice and split from the shell
- 🌍 Built-in rules for CJK, Cyrillic, Greek, accented Latin scripts and emoji – extend them with your own

## Accuracy

Each row of the chart shows a sample text's `o200k_base` token count, the tokenx estimate, and the deviation between them:

<!-- automd:file src="./docs/bench.md" -->

Bars grow left when tokenx underestimates and right when it overestimates. The axis spans the ±10% per-sample deviation bound.

```
                                                          under ◂·▸ over
Team chat transcript (en)               293 →    285          ███│              -2.73%
Vite releases API response            8,075 →  8,551             │██████        +5.89%
tokenx source code                    3,496 →  3,341         ████│              -4.43%
Vite plugin API docs (en)             6,901 →  7,155             │████          +3.68%
Cat article (ja)                     12,437 → 11,914         ████│              -4.21%
Cat article (ko)                      7,117 →  6,815         ████│              -4.24%
Cat article (zh)                      9,057 →  8,953            █│              -1.15%
The Great Gatsby by Fitzgerald (en)   4,391 →  4,479             │██            +2.00%
Die Verwandlung by Kafka (de)         4,437 →  4,384            █│              -1.19%
                                                       ─────────────────────
                                                                        mean     3.28%
```

<!-- /automd -->

Accuracy depends on the kind of text: a holdout corpus of French, English, and Russian documents the heuristics were never calibrated on stays within ±15% per document.

These cases are knowingly left outside the bounds:

- **High-entropy strings** – base64, hashes, digests: ≈-70%.
- **Traditional and classical Chinese** – the hanzi rate is calibrated on contemporary simplified script: ≈-5% to -20%.
- **Scripts without a built-in rule** – Arabic ≈-10% to -30%, Hindi ≈-30%, Hebrew ≈-45%, Thai ≈-60%; a custom language config closes the gap.
- **Chinese or Japanese mixed with Latin words or code** – Latin letters that share a segment with CJK text are priced like hanzi: ≈+30% to +80% for technical prose, up to ≈+200% for a sentence built around code identifiers.

## Installation

```bash
# npm
npm install tokenx

# pnpm
pnpm add tokenx

# yarn
yarn add tokenx
```

On Node.js, tokenx requires version 24 or later.

## Quick Start

```ts
import { estimateTokenCount, isWithinTokenLimit, sliceByTokens, splitByTokens } from 'tokenx'

const text = 'Your text goes here.'

estimateTokenCount(text) // Estimated token count
isWithinTokenLimit(text, 1024) // true
sliceByTokens(text, 0, 5) // Roughly the first five tokens
splitByTokens(text, 100) // Chunks of about 100 tokens each
```

## CLI

The package ships a `tokenx` binary. Run it with `npx tokenx`, or install the package to call `tokenx` directly.

```bash
# Count tokens in a file, a pipe, or several files at once
tokenx README.md
cat article.md | tokenx
tokenx count src/*.ts

# Fail a script when a prompt outgrows its budget (exit code 2)
tokenx prompt.txt --limit 8000

# Extract a token range, or chunk a document for RAG
tokenx slice article.md --end 500
tokenx split article.md --size 500 --overlap 50
```

Only results go to stdout – a bare integer for one input, a table for several, chunks as a JSON array – so `$(tokenx count file.txt)` and `| jq` work unchanged. Notices and errors go to stderr. Run `tokenx --help` for every option.

## API

### Options

Every function takes an optional `options` object last:

- `defaultCharsPerToken` – average characters per token for segments no other rule prices (default: 7)
- `languageConfigs` – `{ pattern, averageCharsPerToken }` entries checked before the built-in language configs; the first matching config prices a segment. They take precedence over every built-in rule except whitespace, the CJK script rates included, and the built-in configs still price whatever no custom config matches.

### `estimateTokenCount(text?, options?)`

Returns the estimated token count of `text`.

```ts
estimateTokenCount('Hello, world!')

estimateTokenCount('Bonjour, ça va très bien ?', {
  defaultCharsPerToken: 4,
  languageConfigs: [
    { pattern: /[éèêëàâîï]/i, averageCharsPerToken: 3 }
  ],
})
```

### `isWithinTokenLimit(text, tokenLimit, options?)`

Returns whether the estimated token count stays within `tokenLimit`, the limit included. Stops counting as soon as the limit is exceeded.

```ts
isWithinTokenLimit('Check this text against a limit', 100) // true
```

### `sliceByTokens(text, start?, end?, options?)`

Extracts a token range like `Array.prototype.slice()`: `start` is inclusive, `end` exclusive, and negative indices count back from the end. A slice can start or end mid-word; adjacent slices still rejoin into the original text.

```ts
const text = 'Hello, world! This is a test sentence.'

sliceByTokens(text, 0, 3) // First three tokens
sliceByTokens(text, -2) // Last two tokens
sliceByTokens(text, 1, -1) // All but the first and last token
```

### `splitByTokens(text, tokensPerChunk, options?)`

Splits text into chunks of about `tokensPerChunk` tokens – for RAG, batch processing, or staying within a context window.

`tokensPerChunk` is a target, not a hard maximum: a chunk closes once it reaches the target, so a single long segment can push a chunk beyond it. Chinese and Japanese text has no spaces, so a whole clause up to its punctuation mark forms one segment, and a long clause can push a chunk well past the target.

The `overlap` option (default: 0, clamped below `tokensPerChunk`) repeats about `overlap` tokens from the end of each chunk at the start of the next. The repeat consists of whole segments, so it runs longer when a segment crosses `overlap` and shorter when one more segment would bring it up to `tokensPerChunk`. A final chunk consisting only of overlap content is never emitted.

```ts
splitByTokens(text, 100)
splitByTokens(text, 100, { overlap: 10 })
```

## License

[MIT](./LICENSE) License © 2023-PRESENT [Johann Schopplich](https://github.com/johannschopplich)