bpe-merge-visualizer/README.md

CLI tool to visualize Byte Pair Encoding (BPE) merge steps

# 🔡 BPE Merge Visualizer

A simple CLI tool to visualize step-by-step Byte Pair Encoding (BPE) merge operations using a mock rule set. Great for learning how BPE tokenization works (like in GPT models).

---

## 🧪 Usage

After running `npm link`, you can run the CLI tool with:

```bash
bpeviz
```

You'll be prompted to enter a word:

```
Enter a word to visualize BPE merging: indivisibility
Step 0: [i] [n] [d] [i] [v] [i] [s] [i] [b] [i] [l] [i] [t] [y]
Step 1: [i] [n] [d] [i] [v] [is] [i] [b] [i] [l] [i] [t] [y]
Step 2: ...
```

---

## 📦 Folder Structure

```
bpe-merge-visualizer/
├── bin/
│   └── cli.js          # CLI entry file
├── src/
│   └── visualizer.js   # Merge logic
├── package.json
└── README.md
```

---

## 🛠 Features

- Step-by-step visualization of BPE merge rules
- Designed to simulate GPT tokenization logic
- Fully extensible to support real `cl100k_base.json` or other vocab

---

## 🔮 Future Improvements

- Support real BPE vocab files
- Add Web/GUI version
- Export to HTML/SVG

---

## 📜 License

MIT

---

## 🤖 Inspired by

- OpenAI's [`tiktoken`](https://github.com/openai/tiktoken)
- HuggingFace's `tokenizers`

---

## 🙌 Author

Made with ❤️ by [MOHD RAZA](https://github.com/raza001)