Modules / Gen AI / BPE Lab

Byte Pair Encoding Tokenizer

Q: Can I train my own tokeniser?

Yes, with tokenizers or SentencePiece. Worth it for a specialised domain or language. And it means the model must be trained or much adapted with it.

Q: How do I count tokens before an API call?

With the model's own tokeniser — tiktoken for OpenAI models, the transformers tokeniser if not. Estimating from word count is unreliable.

Generative AI & LLMsBy Ashish JangraUpdated 30 September 2026

Train a real BPE tokenizer in your browser. Start from single characters, repeatedly merge the most frequent adjacent pair, and watch a subword vocabulary build itself from the data.

1. Training Corpus

Number of Merges 20

2. Tokenize a Word

Any string can be encoded — unknown words simply fall back to characters, which is why BPE never produces an [UNK].

Vocabulary Statistics

Base Chars0

Merges Learned0

Vocab Size0

Tokens / Word0

Segmentation of “”

How it got there

Learned Merge Rules (in order)

RANK = PRIORITY

Overview

Train a real BPE tokenizer in your browser.

In this guide

The Two Failure Modes It Avoids
The Algorithm, in Four Lines
Merge Order Is the Model
Building a vocabulary by merging
What it gives on real text
Byte-level BPE, and why it matters
Learning a vocabulary by merging the commonest pair
Tests to try
Where that leaves you
The variants
What tokenisation decides for the model
Questions people ask
Recap in one screen

Byte Pair Encoding: Building a Vocabulary from Data

Before a language model sees a single word, a tokenizer must turn text into integers. Byte Pair Encoding is the algorithm most modern LLMs use, and it solves a problem that both word-level and character-level tokenizers get badly wrong.

The Two Failure Modes It Avoids

Word-level: each distinct word needs its own entry. The vocabulary explodes into the millions, and any word missing from it turns a useless [UNK]. Each typo, name and new term is lost.
Character-level: a tiny vocabulary that can spell a thing, but sequences turn enormously long and the model must relearn that c-a-t means a thing.

BPE lands in between: frequent words turn single tokens, rare words split into meaningful pieces, and nothing is ever out-of-vocabulary. The algorithm starts with the character-level vocabulary (plus a special end-of-word marker) and iteratively merges the most frequent next pair into a new token. Each merge reduces the total sequence length by the number of instances of that pair. After a fixed number of merges (the vocabulary size minus the initial character set), the vocabulary is complete. GPT-2 uses 50,257 tokens; GPT-4 uses around 100,000. A larger vocabulary means shorter sequences (fewer tokens per sentence, so faster inference and a longer effective context window) but a larger embedding table. The merge order is the tokenizer — at inference time, it applies the same merges in the same rank to any new text, greedily splitting unknown words into the longest known pieces. This page runs BPE step by step on a sample corpus, shows the merge table growing after each iteration, and covers the vocabulary-size tradeoff that decides how many merges to perform.

The Algorithm, in Four Lines

Split each word into characters, marking the word end (shown here as </w>).
Count each next symbol pair across the whole corpus.
Merge the single most frequent pair all over it occurs, and record that merge.
Repeat until you have the number of merges you asked for.

That is the whole method. Vocabulary size is simply base characters plus the number of merges — which is why it is an exact dial. Rather than a thing you discover after the fact.

Merge Order Is the Model

The merge list is ordered, and that order matters at encoding time. To tokenize a new word you replay the merges from rank 1 downward, applying each wherever it fits. The "How it got there" panel shows this replay step by step. The word starts as loose characters and step by step fuses.

Notice that the earliest merges are almost always very common fragments (e+s, t+h). While later merges assemble whole frequent words. The vocabulary discovers morphology — prefixes, suffixes, stems — without ever being told that such things exist.

Building a vocabulary by merging

Byte-pair encoding starts from single characters and repeatedly merges the most frequent next pair, recording each merge as a new vocabulary entry.

Worked through on a tiny corpus — the words "low", "lower", "newest", "widest", with the counts they appear:

Start with characters: l o w e r n s t i d
The most frequent next pair is e + s → merge into es
Next most frequent: es + t → est
Next: l + o → lo
Next: lo + w → low

Continue until the vocabulary reaches the target size — 30,000 or 100,000 entries in practice. The merges are recorded in order, and applying them in that same order is how new text is tokenised.

The outcome is a vocabulary that spends its budget where the text really is. Frequent words turn single tokens; rare words remain as fragments; and. This is because the single characters (or bytes) are always in the vocabulary, nothing is ever unrepresentable.

That last property is why the <UNK> token has in effect disappeared from modern models.

What it gives on real text

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("gpt2")

tok.tokenize("tokenization")        # ['token', 'ization']
tok.tokenize("Tokenization")        # ['Token', 'ization']  -- case matters
tok.tokenize("antidisestablishmentarianism")
# ['ant', 'id', 'ises', 't', 'ablish', 'ment', 'arian', 'ism']
tok.tokenize(" hello")              # ['Ġhello']  -- Ġ marks a leading space

Three observations from those lines.

Common words are single tokens. Long or rare ones fragment into pieces that were frequent in the training corpus.

Leading spaces are part of the token. GPT-style tokenisers treat " hello" and "hello" as other tokens. This is why a prompt ending in a space can change the output clearly.

Case matters. "Token" and "token" are other entries, so capitalisation consumes vocabulary and affects tokenisation.

Byte-level BPE, and why it matters

Plain BPE over characters has a gap: a character never seen in training cannot be shown. With Unicode's 149,000+ code points, that gap is real.

Byte-level BPE runs on UTF-8 bytes rather than characters. There are only 256 doable bytes, all of which are in the base vocabulary. So any text in any script — plus emoji, plus binary noise — is representable.

The cost is speed for non-Latin scripts. A Chinese character is three UTF-8 bytes, so it costs at least one token and often more; the same sentence in Chinese and in English can differ severalfold in token count.

Text	Approximate tokens
English prose	~1.3 per word
Code	~2 per word, more with symbols
Chinese / Japanese	~1 per character
Cyrillic, Arabic	~2–3 per word
Emoji	1–4 each

That table is a real cost and latency gap when calling an API. And it is why multilingual models often use larger vocabularies to compensate.

Learning a vocabulary by merging the commonest pair

BPE starts from single characters and repeatedly merges whichever next pair is commonest, until the vocabulary is the size you asked for. This trains one on a small corpus, tokenises with it, and shows why token counts are the thing that really costs you money.

example_01.pyNumPy

import numpy as np
from collections import Counter

CORPUS = ("low low low low low lower lower newest newest newest newest "
          "newest newest widest widest widest")

# each word is a tuple of symbols; </w> marks the end so 'low' and the
# 'low' inside 'lower' can be told apart
words = Counter(CORPUS.split())
vocab = {tuple(list(w) + ["</w>"]): c for w, c in words.items()}

print("THE TRAINING CORPUS, as word counts:")
for w, c in sorted(words.items(), key=lambda kv: -kv[1]):
    print("   %-10s x%d" % (w, c))
print()
print("EVERY WORD STARTS AS SINGLE CHARACTERS. </w> marks a word boundary,")
print("so the model can tell 'low' the word from 'low' inside 'lower':")
for w, c in list(vocab.items())[:3]:
    print("   %s" % " ".join(w))
print()

def pair_counts(vocab):
    p = Counter()
    for word, c in vocab.items():
        for i in range(len(word) - 1):
            p[(word[i], word[i + 1])] += c
    return p

Output

Building on the same setup, the next block shows the left cases.

example_02.pyNumPy

import numpy as np
from collections import Counter
CORPUS = ("low low low low low lower lower newest newest newest newest "
          "newest newest widest widest widest")

def merge(vocab, pair):
    out = {}
    for word, c in vocab.items():
        w, i = [], 0
        while i < len(word):
            if i < len(word) - 1 and (word[i], word[i + 1]) == pair:
                w.append(word[i] + word[i + 1])
                i += 2
            else:
                w.append(word[i])
                i += 1
        out[tuple(w)] = c
    return out
print("done")

Output

The below block covers the left steps.

example_03.pyNumPy

import numpy as np
from collections import Counter
CORPUS = ("low low low low low lower lower newest newest newest newest "
          "newest newest widest widest widest")

MERGES = 10
learned = []
print("NOW MERGE THE COMMONEST ADJACENT PAIR, %d times:" % MERGES)
print("%-8s %-16s %8s %s" % ("step", "pair merged", "count", "runners-up"))
for step in range(MERGES):
    p = pair_counts(vocab)
    if not p:
        break
    best, cnt = p.most_common(1)[0]
    rest = ", ".join("%s%s:%d" % (a, b, n) for (a, b), n in p.most_common(4)[1:])
    learned.append(best)
    print("%-8d %-16s %8d %s" % (step + 1, "'%s' + '%s'" % best, cnt, rest))
    vocab = merge(vocab, best)
print()
print("THE WORDS NOW, after those merges:")
for word, c in sorted(vocab.items(), key=lambda kv: -kv[1]):
    print("   %-30s x%d  (%d token%s)"
          % (" | ".join(word), c, len(word), "" if len(word) == 1 else "s"))
print("   the two commonest words collapsed to a SINGLE token each. the")
print("   rarer ones did not, and that is the entire design: BPE spends")
print("   its vocabulary budget where the text actually is.")
print("   'lower' is the interesting row. it appears only twice, so no")
print("   merge ever fired for its ending -- and it still costs 4 tokens")
print("   even though 'low' became a token at step 5. the merges are")
print("   learned in frequency order, and a word that never wins a round")
print("   simply never gets shorter.")
print()

Output

The next block goes on with the left demonstrations.

example_04.pyNumPy

import numpy as np
from collections import Counter
CORPUS = ("low low low low low lower lower newest newest newest newest "
          "newest newest widest widest widest")

print("THE LEARNED MERGES, IN ORDER. this ordered list IS the tokenizer --")
print("applying it to new text is just replaying these rules:")
for i, (a, b) in enumerate(learned):
    print("   %2d. %s + %s -> %s" % (i + 1, a, b, a + b))
print()

def tokenize(word):
    sym = list(word) + ["</w>"]
    for a, b in learned:                       # in the order they were learned
        i = 0
        while i < len(sym) - 1:
            if sym[i] == a and sym[i + 1] == b:
                sym[i:i + 2] = [a + b]
            else:
                i += 1
    return sym

print("TOKENISING NEW WORDS with those rules:")
print("%-14s %8s   %s" % ("word", "tokens", "split"))
for w in ["low", "lower", "lowest", "newest", "wider", "slowest", "zebra"]:
    t = tokenize(w)
    print("%-14s %8d   %s" % (w, len(t), " | ".join(t)))
print("   'low' and 'newest' were in the training text and cost few tokens.")
print("   'zebra' was not, and falls apart into characters -- but it is")
print("   still representable. THAT is why BPE has no unknown token: the")
print("   base vocabulary is every single character, so any string can")
print("   always be spelled out, however badly.")
print()

Output

Building on the same setup, the next block shows the left cases.

example_05.pyNumPy

import numpy as np
from collections import Counter
CORPUS = ("low low low low low lower lower newest newest newest newest "
          "newest newest widest widest widest")

print("AND THAT COSTS REAL MONEY, because you are billed per token:")
SAMPLES = [("common English", "low lower newest widest"),
           ("rare words", "zebra xylophone quixotic"),
           ("a made-up identifier", "getUserByIdAsync")]
print("%-26s %10s %10s %14s"
      % ("text", "chars", "tokens", "chars/token"))
for label, text in SAMPLES:
    toks = [t for w in text.split() for t in tokenize(w)]
    nchar = len(text.replace(" ", ""))
    print("%-26s %10d %10d %14.2f"
          % (label, nchar, len(toks), nchar / float(len(toks))))
print("   the same number of CHARACTERS costs very different numbers of")
print("   TOKENS depending on whether the tokenizer has seen that kind of")
print("   text before. this is not a curiosity:")
print("      - a language under-represented in training costs several")
print("        times more per sentence than English, for identical")
print("        meaning, and fits several times less into a context window")
print("      - code, JSON and base64 tokenise badly for the same reason")
print("      - counting characters or words to estimate cost or context")

Output

The below block covers the left steps.

example_06.pyNumPy

print("        usage will be wrong, sometimes by a factor of 3")
print()

print("WHY BPE RATHER THAN WORDS OR CHARACTERS:")
print("%-22s %14s %-22s %s"
      % ("scheme", "vocab size", "unknown words", "sequence length"))
for row in (("characters", "~100", "impossible", "very long"),
            ("whole words", "500k+", "constant problem", "short"),
            ("BPE / subword", "30k-100k", "impossible", "in between")):
    print("%-22s %14s %-22s %s" % row)
print("   whole-word vocabularies need an <UNK> token for everything they")
print("   have not seen, and <UNK> destroys information permanently -- a")
print("   model cannot reason about a word it was handed as 'unknown'.")
print("   character models never have that problem but need far more")
print("   steps per sentence, and attention is quadratic in that length.")
print("   BPE sits between: frequent things are one token, everything else")
print("   degrades gracefully into pieces, and nothing is ever unknown.")
print()
print("one last property worth knowing: the merge list is ORDERED and must")
print("be replayed in order. the tokenizer is not a dictionary lookup, it")
print("is a deterministic replay of the training procedure -- which is why")
print("a model and its tokenizer are inseparable, and why using the wrong")
print("tokenizer with a model produces fluent nonsense rather than an")
print("error.")

Output

Tests to try

Set merges to 0. Each word shatters into single characters. The tokens-per-word figure is at its worst.
Drag the slider up slowly and watch "Tokens / Word" fall. Each merge buys compression, with the earliest merges buying the most.
Tokenize lowest after training on the repetitive sample. It often splits into a stem plus est. A suffix the algorithm found purely from count.
Now type a word the corpus has never seen, such as zyxwv. It falls back to characters instead of failing. This graceful degradation is BPE's most key property.
Switch to the code-like sample. Other merges appear fully,. This is because the vocabulary is a direct fingerprint of the training data —. This is just why a model tokenizes unfamiliar domains slowly.

Where that leaves you

BPE is a compression algorithm repurposed as a vocabulary builder. Merge what is frequent, leave the rest in pieces. This lab runs the genuine training loop on any corpus you paste in. The merge table you see is the actual product of counting pairs, not a stored case. Real tokenizers add byte-level fallbacks and pre-tokenization rules, but the core loop is just this.

The variants

Algorithm	Merge criterion	Used by
BPE	Most frequent adjacent pair	GPT-2, GPT-3, Llama
WordPiece	Pair that most increases corpus likelihood	BERT, DistilBERT
Unigram	Start large, prune the least useful tokens	T5, XLNet, mBART
SentencePiece	A wrapper implementing BPE or Unigram on raw text	T5, Llama, many multilingual models

WordPiece differs from BPE only in the merge rule:. Rather than raw count, it picks the merge that most improves the likelihood of the corpus under a unigram model. In practice the vocabularies are close. It marks continuations with ## — "token", "##ization".

Unigram works backwards: begin with a large candidate vocabulary and iteratively remove the tokens whose loss hurts least. It can give a few valid segmentations of the same word with chances. This enables subword regularisation during training.

SentencePiece is worth grasp as a other kind of thing — not an algorithm but an implementation that treats input as a raw byte stream with no pre-tokenisation on whitespace. That matters for languages that do not separate words with spaces. And it makes detokenisation just reversible.

What tokenisation decides for the model

A few well-known model behaviours trace right back to this step.

Poor spelling and character counting. "strawberry" may be two or three tokens, none of which is a letter. Asking how many r's it holds asks the model to reason about a thing it cannot see.

Inconsistent arithmetic. "1234" might be one token or a few depending on the tokeniser and the nearby text. So digits are not always shown. Some newer models tokenise digits one by one to improve this.

Context limits in tokens, not words. A 128,000-token window is roughly 96,000 English words, and far fewer for code or other scripts.

Response to trailing spaces. Because the space is part of the token, a prompt ending with a space tokenises else from one that does not.

The tokeniser must ship with the model. Token id 1,547 means one thing in one vocabulary and a thing else in one more. A mismatch gives fluent nonsense rather than an error.

Questions people ask

How large should the vocabulary be? 30,000–50,000 for a single language; 100,000–250,000 for multilingual models. Larger means shorter sequences and a bigger embedding table.

Can I train my own tokeniser? Yes, with tokenizers or SentencePiece. Worth it for a specialised domain or language. And it means the model must be trained or much adapted with it.

Why does my word split oddly? Because those fragments were the frequent merges in the training corpus. It looks arbitrary and is fully determined by corpus stats.

Is BPE reversible? Yes — concatenating the tokens reconstructs the first text just. This is why byte-level variants are careful about spaces.

Do I need to lowercase first? No, for a cased tokeniser. It was trained on cased text and lowercasing loses data.

How do I count tokens before an API call? With the model's own tokeniser — tiktoken for OpenAI models, the transformers tokeniser if not. Estimating from word count is unreliable.

Recap in one screen

BPE starts from characters and repeatedly merges the most frequent next pair until the vocabulary is full.
Frequent words turn single tokens; rare ones fragment; characters remain as a fallback. So nothing is unknown.
Byte-level BPE works on UTF-8 bytes, so any text is representable. At more tokens for non-Latin scripts.
WordPiece, Unigram and SentencePiece are changes on the same idea with other merge or pruning rules.
Tokenisation explains poor spelling, inconsistent arithmetic, token-based context limits and trailing-space response.

Recall check

0 of 3

Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.

What does this module say about “The Two Failure Modes It Avoids”?
What does this module say about “The Algorithm, in Four Lines”?
What does this module say about “Merge Order Is the Model”?

Cheat sheet