Train a real BPE tokenizer in your browser. Start from single characters, repeatedly merge the most frequent adjacent pair, and watch a subword vocabulary build itself from the data.
Overview
The Two Failure Modes It Avoids
Word-level: every distinct word needs its own entry. The vocabulary explodes into the millions, and any word missing from it becomes a useless [UNK] — every typo, name and new term is lost.
Character-level: a tiny vocabulary that can spell anything, but sequences become enormously long and the model must relearn that c-a-t means something.
BPE lands in between: frequent words become single tokens, rare words split into meaningful pieces, and nothing is ever out-of-vocabulary.
Segmentation of “”
How it got there
Learned Merge Rules (in order)
RANK = PRIORITY
Byte Pair Encoding: Building a Vocabulary from Data
Before a language model sees a single word, a tokenizer must turn text into integers. Byte Pair Encoding is the algorithm most modern LLMs use, and it solves a problem that both word-level and character-level tokenizers get badly wrong.
The Algorithm, in Four Lines
Split every word into characters, marking the word end (shown here as </w>).
Count every adjacent symbol pair across the whole corpus.
Merge the single most frequent pair everywhere it occurs, and record that merge.
Repeat until you have the number of merges you asked for.
That is the entire method. Vocabulary size is simply base characters plus the number of merges — which is why it is an exact dial rather than something you discover after the fact.
Merge Order Is the Model
The merge list is ordered, and that order matters at encoding time: to tokenize a new word you replay the merges from rank 1 downward, applying each wherever it fits. The "How it got there" panel shows this replay step by step — the word starts as loose characters and progressively fuses.
Notice that the earliest merges are almost always extremely common fragments (e+s, t+h), while later merges assemble whole frequent words. The vocabulary discovers morphology — prefixes, suffixes, stems — without ever being told that such things exist.
Building a vocabulary by merging
Byte-pair encoding starts from individual characters and repeatedly merges the most frequent adjacent pair, recording each merge as a new vocabulary entry.
Worked through on a tiny corpus — the words "low", "lower", "newest", "widest", with the counts they appear:
Start with characters: l o w e r n s t i d
The most frequent adjacent pair is e + s → merge into es
Next most frequent: es + t → est
Next: l + o → lo
Next: lo + w → low
Continue until the vocabulary reaches the target size — 30,000 or 100,000 entries in practice. The merges are recorded in order, and applying them in that same order is how new text is tokenised.
The outcome is a vocabulary that spends its budget where the text actually is. Frequent words become single tokens; rare words remain as fragments; and because the individual characters (or bytes) are always in the vocabulary, nothing is ever unrepresentable.
That last property is why the <UNK> token has essentially disappeared from modern models.
What it produces on real text
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("gpt2")
tok.tokenize("tokenization") # ['token', 'ization']
tok.tokenize("Tokenization") # ['Token', 'ization'] -- case matters
tok.tokenize("antidisestablishmentarianism")
# ['ant', 'id', 'ises', 't', 'ablish', 'ment', 'arian', 'ism']
tok.tokenize(" hello") # ['Ġhello'] -- Ġ marks a leading space
Three observations from those lines.
Common words are single tokens; long or rare ones fragment into pieces that were frequent in the training corpus.
Leading spaces are part of the token. GPT-style tokenisers treat " hello" and "hello" as different tokens, which is why a prompt ending in a space can change the output noticeably.
Case matters. "Token" and "token" are different entries, so capitalisation consumes vocabulary and affects tokenisation.
Byte-level BPE, and why it matters
Plain BPE over characters has a gap: a character never seen in training cannot be represented. With Unicode's 149,000+ code points, that gap is real.
Byte-level BPE operates on UTF-8 bytes rather than characters. There are only 256 possible bytes, all of which are in the base vocabulary, so any text in any script — plus emoji, plus binary noise — is representable.
The cost is efficiency for non-Latin scripts. A Chinese character is three UTF-8 bytes, so it costs at least one token and often more; the same sentence in Chinese and in English can differ severalfold in token count.
Text
Approximate tokens
English prose
~1.3 per word
Code
~2 per word, more with symbols
Chinese / Japanese
~1 per character
Cyrillic, Arabic
~2–3 per word
Emoji
1–4 each
That table is a real cost and latency difference when calling an API, and it is why multilingual models often use larger vocabularies to compensate.
Learning a vocabulary by merging the commonest pair
BPE starts from single characters and repeatedly merges whichever adjacent pair is commonest, until the vocabulary is the size you asked for. This trains one on a small corpus, tokenises with it, and shows why token counts are the thing that actually costs you money.
example_01.pyNumPy
import numpy as np
from collections import Counter
CORPUS = ("low low low low low lower lower newest newest newest newest "
"newest newest widest widest widest")
# each word is a tuple of symbols; </w> marks the end so 'low' and the
# 'low' inside 'lower' can be told apart
words = Counter(CORPUS.split())
vocab = {tuple(list(w) + ["</w>"]): c for w, c in words.items()}
print("THE TRAINING CORPUS, as word counts:")
for w, c in sorted(words.items(), key=lambda kv: -kv[1]):
print(" %-10s x%d" % (w, c))
print()
print("EVERY WORD STARTS AS SINGLE CHARACTERS. </w> marks a word boundary,")
print("so the model can tell 'low' the word from 'low' inside 'lower':")
for w, c in list(vocab.items())[:3]:
print(" %s" % " ".join(w))
print()
def pair_counts(vocab):
p = Counter()
for word, c in vocab.items():
for i in range(len(word) - 1):
p[(word[i], word[i + 1])] += c
return p
def merge(vocab, pair):
out = {}
for word, c in vocab.items():
w, i = [], 0
while i < len(word):
if i < len(word) - 1 and (word[i], word[i + 1]) == pair:
w.append(word[i] + word[i + 1])
i += 2
else:
w.append(word[i])
i += 1
out[tuple(w)] = c
return out
MERGES = 10
learned = []
print("NOW MERGE THE COMMONEST ADJACENT PAIR, %d times:" % MERGES)
print("%-8s %-16s %8s %s" % ("step", "pair merged", "count", "runners-up"))
for step in range(MERGES):
p = pair_counts(vocab)
if not p:
break
best, cnt = p.most_common(1)[0]
rest = ", ".join("%s%s:%d" % (a, b, n) for (a, b), n in p.most_common(4)[1:])
learned.append(best)
print("%-8d %-16s %8d %s" % (step + 1, "'%s' + '%s'" % best, cnt, rest))
vocab = merge(vocab, best)
print()
print("THE WORDS NOW, after those merges:")
for word, c in sorted(vocab.items(), key=lambda kv: -kv[1]):
print(" %-30s x%d (%d token%s)"
% (" | ".join(word), c, len(word), "" if len(word) == 1 else "s"))
print(" the two commonest words collapsed to a SINGLE token each. the")
print(" rarer ones did not, and that is the entire design: BPE spends")
print(" its vocabulary budget where the text actually is.")
print(" 'lower' is the interesting row. it appears only twice, so no")
print(" merge ever fired for its ending -- and it still costs 4 tokens")
print(" even though 'low' became a token at step 5. the merges are")
print(" learned in frequency order, and a word that never wins a round")
print(" simply never gets shorter.")
print()
print("THE LEARNED MERGES, IN ORDER. this ordered list IS the tokenizer --")
print("applying it to new text is just replaying these rules:")
for i, (a, b) in enumerate(learned):
print(" %2d. %s + %s -> %s" % (i + 1, a, b, a + b))
print()
def tokenize(word):
sym = list(word) + ["</w>"]
for a, b in learned: # in the order they were learned
i = 0
while i < len(sym) - 1:
if sym[i] == a and sym[i + 1] == b:
sym[i:i + 2] = [a + b]
else:
i += 1
return sym
print("TOKENISING NEW WORDS with those rules:")
print("%-14s %8s %s" % ("word", "tokens", "split"))
for w in ["low", "lower", "lowest", "newest", "wider", "slowest", "zebra"]:
t = tokenize(w)
print("%-14s %8d %s" % (w, len(t), " | ".join(t)))
print(" 'low' and 'newest' were in the training text and cost few tokens.")
print(" 'zebra' was not, and falls apart into characters -- but it is")
print(" still representable. THAT is why BPE has no unknown token: the")
print(" base vocabulary is every single character, so any string can")
print(" always be spelled out, however badly.")
print()
print("AND THAT COSTS REAL MONEY, because you are billed per token:")
SAMPLES = [("common English", "low lower newest widest"),
("rare words", "zebra xylophone quixotic"),
("a made-up identifier", "getUserByIdAsync")]
print("%-26s %10s %10s %14s"
% ("text", "chars", "tokens", "chars/token"))
for label, text in SAMPLES:
toks = [t for w in text.split() for t in tokenize(w)]
nchar = len(text.replace(" ", ""))
print("%-26s %10d %10d %14.2f"
% (label, nchar, len(toks), nchar / float(len(toks))))
print(" the same number of CHARACTERS costs very different numbers of")
print(" TOKENS depending on whether the tokenizer has seen that kind of")
print(" text before. this is not a curiosity:")
print(" - a language under-represented in training costs several")
print(" times more per sentence than English, for identical")
print(" meaning, and fits several times less into a context window")
print(" - code, JSON and base64 tokenise badly for the same reason")
print(" - counting characters or words to estimate cost or context")
print(" usage will be wrong, sometimes by a factor of 3")
print()
print("WHY BPE RATHER THAN WORDS OR CHARACTERS:")
print("%-22s %14s %-22s %s"
% ("scheme", "vocab size", "unknown words", "sequence length"))
for row in (("characters", "~100", "impossible", "very long"),
("whole words", "500k+", "constant problem", "short"),
("BPE / subword", "30k-100k", "impossible", "in between")):
print("%-22s %14s %-22s %s" % row)
print(" whole-word vocabularies need an <UNK> token for everything they")
print(" have not seen, and <UNK> destroys information permanently -- a")
print(" model cannot reason about a word it was handed as 'unknown'.")
print(" character models never have that problem but need far more")
print(" steps per sentence, and attention is quadratic in that length.")
print(" BPE sits between: frequent things are one token, everything else")
print(" degrades gracefully into pieces, and nothing is ever unknown.")
print()
print("one last property worth knowing: the merge list is ORDERED and must")
print("be replayed in order. the tokenizer is not a dictionary lookup, it")
print("is a deterministic replay of the training procedure -- which is why")
print("a model and its tokenizer are inseparable, and why using the wrong")
print("tokenizer with a model produces fluent nonsense rather than an")
print("error.")
Output
Experiments to try
Set merges to 0. Every word shatters into individual characters — the tokens-per-word figure is at its worst.
Drag the slider up slowly and watch "Tokens / Word" fall. Each merge buys compression, with the earliest merges buying the most.
Tokenize lowest after training on the repetitive sample. It typically splits into a stem plus est — a suffix the algorithm found purely from frequency.
Now type a word the corpus has never seen, such as zyxwv. It falls back to characters instead of failing — this graceful degradation is BPE's most important property.
Switch to the code-like sample. Different merges appear entirely, because the vocabulary is a direct fingerprint of the training data — which is exactly why a model tokenizes unfamiliar domains inefficiently.
Where that leaves you
BPE is a compression algorithm repurposed as a vocabulary builder: merge what is frequent, leave the rest in pieces. This lab runs the genuine training loop on whatever corpus you paste in — the merge table you see is the actual product of counting pairs, not a stored example. Real tokenizers add byte-level fallbacks and pre-tokenization rules, but the core loop is exactly this.
The variants
Algorithm
Merge criterion
Used by
BPE
Most frequent adjacent pair
GPT-2, GPT-3, Llama
WordPiece
Pair that most increases corpus likelihood
BERT, DistilBERT
Unigram
Start large, prune the least useful tokens
T5, XLNet, mBART
SentencePiece
A wrapper implementing BPE or Unigram on raw text
T5, Llama, many multilingual models
WordPiece differs from BPE only in the merge criterion: rather than raw frequency, it picks the merge that most improves the likelihood of the corpus under a unigram model. In practice the vocabularies are similar. It marks continuations with ## — "token", "##ization".
Unigram works backwards: begin with a large candidate vocabulary and iteratively remove the tokens whose loss hurts least. It can produce several valid segmentations of the same word with probabilities, which enables subword regularisation during training.
SentencePiece is worth understanding as a different kind of thing — not an algorithm but an implementation that treats input as a raw byte stream with no pre-tokenisation on whitespace. That matters for languages that do not separate words with spaces, and it makes detokenisation exactly reversible.
What tokenisation decides for the model
Several well-known model behaviours trace directly back to this step.
Poor spelling and character counting. "strawberry" may be two or three tokens, none of which is a letter. Asking how many r's it contains asks the model to reason about something it cannot see.
Inconsistent arithmetic. "1234" might be one token or several depending on the tokeniser and the surrounding text, so digits are not consistently represented. Some newer models tokenise digits individually to improve this.
Context limits in tokens, not words. A 128,000-token window is roughly 96,000 English words, and far fewer for code or other scripts.
Sensitivity to trailing spaces. Because the space is part of the token, a prompt ending with a space tokenises differently from one that does not.
The tokeniser must ship with the model. Token id 1,547 means one thing in one vocabulary and something else in another. A mismatch produces fluent nonsense rather than an error.
Questions people ask
How large should the vocabulary be? 30,000–50,000 for a single language; 100,000–250,000 for multilingual models. Larger means shorter sequences and a bigger embedding table.
Can I train my own tokeniser? Yes, with tokenizers or SentencePiece — worth it for a specialised domain or language, and it means the model must be trained or substantially adapted with it.
Why does my word split oddly? Because those fragments were the frequent merges in the training corpus. It looks arbitrary and is entirely determined by corpus statistics.
Is BPE reversible? Yes — concatenating the tokens reconstructs the original text exactly, which is why byte-level variants are careful about spaces.
Do I need to lowercase first? No, for a cased tokeniser. It was trained on cased text and lowercasing loses information.
How do I count tokens before an API call? With the model's own tokeniser — tiktoken for OpenAI models, the transformers tokeniser otherwise. Estimating from word count is unreliable.
Recap in one screen
BPE starts from characters and repeatedly merges the most frequent adjacent pair until the vocabulary is full.
Frequent words become single tokens; rare ones fragment; characters remain as a fallback, so nothing is unknown.
Byte-level BPE works on UTF-8 bytes, so any text is representable — at more tokens for non-Latin scripts.
WordPiece, Unigram and SentencePiece are variations on the same idea with different merge or pruning criteria.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “The Two Failure Modes It Avoids”?
BPE lands in between: frequent words become single tokens, rare words split into meaningful pieces, and nothing is ever out-of-vocabulary.
What does this module say about “The Algorithm, in Four Lines”?
That is the entire method. Vocabulary size is simply base characters plus the number of merges — which is why it is an exact dial rather than something you discover after the fact.
What does this module say about “Merge Order Is the Model”?
The merge list is ordered , and that order matters at encoding time: to tokenize a new word you replay the merges from rank 1 downward, applying each wherever it fits. The "How it got there" panel shows this replay step by step — the word starts as loose characters and progressively fuses.
Cheat sheet
Byte Pair Encoding Tokenizer
Train a real BPE tokenizer in your browser. Start from single characters, repeatedly merge the most frequent adjacent pair, and watch a subword vocabulary build itself from the data.
GEN AI · vizlearn.in/gen_ai/byte_pair_encoding_tokenizer.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.