Modules / Gen AI / BPE Lab

Byte Pair Encoding Tokenizer

Train a real BPE tokenizer in your browser. Start from single characters, repeatedly merge the most frequent adjacent pair, and watch a subword vocabulary build itself from the data.

1. Training Corpus

Number of Merges 20

2. Tokenize a Word

Any string can be encoded — unknown words simply fall back to characters, which is why BPE never produces an [UNK].

Vocabulary Statistics

Base Chars0

Merges Learned0

Vocab Size0

Tokens / Word0

Segmentation of “”

How it got there

Learned Merge Rules (in order)

RANK = PRIORITY

Byte Pair Encoding: Building a Vocabulary from Data

Before a language model sees a single word, a tokenizer must turn text into integers. Byte Pair Encoding is the algorithm most modern LLMs use, and it solves a problem that both word-level and character-level tokenizers get badly wrong.

The Two Failure Modes It Avoids

Word-level: every distinct word needs its own entry. The vocabulary explodes into the millions, and any word missing from it becomes a useless [UNK] — every typo, name and new term is lost.
Character-level: a tiny vocabulary that can spell anything, but sequences become enormously long and the model must relearn that c-a-t means something.

BPE lands in between: frequent words become single tokens, rare words split into meaningful pieces, and nothing is ever out-of-vocabulary.

The Algorithm, in Four Lines

Split every word into characters, marking the word end (shown here as </w>).
Count every adjacent symbol pair across the whole corpus.
Merge the single most frequent pair everywhere it occurs, and record that merge.
Repeat until you have the number of merges you asked for.

That is the entire method. Vocabulary size is simply base characters plus the number of merges — which is why it is an exact dial rather than something you discover after the fact.

Merge Order Is the Model

The merge list is ordered, and that order matters at encoding time: to tokenize a new word you replay the merges from rank 1 downward, applying each wherever it fits. The "How it got there" panel shows this replay step by step — the word starts as loose characters and progressively fuses.

Notice that the earliest merges are almost always extremely common fragments (e+s, t+h), while later merges assemble whole frequent words. The vocabulary discovers morphology — prefixes, suffixes, stems — without ever being told that such things exist.

Interactive Exploration Guide

Set merges to 0. Every word shatters into individual characters — the tokens-per-word figure is at its worst.
Drag the slider up slowly and watch "Tokens / Word" fall. Each merge buys compression, with the earliest merges buying the most.
Tokenize lowest after training on the repetitive sample. It typically splits into a stem plus est — a suffix the algorithm found purely from frequency.
Now type a word the corpus has never seen, such as zyxwv. It falls back to characters instead of failing — this graceful degradation is BPE's most important property.
Switch to the code-like sample. Different merges appear entirely, because the vocabulary is a direct fingerprint of the training data — which is exactly why a model tokenizes unfamiliar domains inefficiently.

Key Takeaway

BPE is a compression algorithm repurposed as a vocabulary builder: merge what is frequent, leave the rest in pieces. This lab runs the genuine training loop on whatever corpus you paste in — the merge table you see is the actual product of counting pairs, not a stored example. Real tokenizers add byte-level fallbacks and pre-tokenization rules, but the core loop is exactly this.

Cheat sheet