Break down raw text into meaningful units (tokens) using various segmentation strategies.
Overview
What is Tokenization?
Tokenization is the fundamental first step in any Natural Language Processing (NLP) pipeline. It's the process of breaking down a stream of raw text into smaller, meaningful units called tokens. These tokens can be words, characters, or sub-word units, depending on the chosen strategy. For a computer to understand human language, it must first dissect it into these discrete pieces, much like how we learn to read by first identifying individual words.
Input Corpus
Tokenization Method
Whitespace
Split by spaces only
Word (Regex)
Split words & punctuation
Sentence
Split by delimiters (. ! ?)
Character
Split every character
Token Stream
0 Tokens0 Unique
Click 'Tokenize' to break the text apart.
A Beginner's Guide to Text Tokenization
Learn how to break down text into meaningful units for NLP models.
1. Why is Tokenization So Important?
Imagine trying to understand a sentence by looking at it as one continuous string of letters. It would be nearly impossible. Tokenization provides the structure that machines need. By converting a sentence like "NLP is fascinating!" into tokens such as ["NLP", "is", "fascinating", "!"], we create a list of items that a model can count, analyze, and assign meaning to. This process is the gateway to almost every NLP task, including sentiment analysis, machine translation, and text summarization.
2. Exploring Different Tokenization Methods
There's no single "best" way to tokenize text; the right method depends on the task and the language. The interactive visualizer above lets you experiment with the most common strategies. Let's explore them:
Whitespace Tokenization: This is the simplest method. It splits the text based on empty spaces. While fast and straightforward, it's often too basic. For example, it would treat "New York" as two separate tokens, ["New", "York"], and might fail to separate a word from punctuation, like in "fun!".
Word (Regex) Tokenization: A more sophisticated approach that uses regular expressions (regex) to define what constitutes a word. This method is much better at handling punctuation and special cases. For instance, it can correctly separate "fun!" into ["fun", "!"] and can be configured to handle hyphenated words or contractions like "don't".
Sentence Tokenization: Instead of words, this method splits the text into individual sentences. It identifies sentence boundaries using delimiters like periods (.), question marks (?), and exclamation marks (!). This is a crucial preprocessing step for tasks that need to understand the context of a full sentence, such as document summarization.
Character Tokenization: This method breaks the text down to its most granular level: individual characters. The sentence "Hello" becomes ["H", "e", "l", "l", "o"]. This approach is useful for languages without clear word boundaries or for tasks like spell-checking and text generation, as it creates a very small, manageable vocabulary.
Cutting text into pieces a model can count
A model cannot consume a string. It consumes integers, each pointing at a row of an embedding table. Tokenisation is the step that decides what those integers stand for, and the choice shapes everything downstream.
Three levels are possible:
Level
"unhappiness" becomes
Vocabulary size
Character
u, n, h, a, p, ...
~100
Word
unhappiness
100,000+
Subword
un, happi, ness
30,000–100,000
Character-level needs a tiny vocabulary and never meets an unknown symbol, but sequences become very long and the model has to learn spelling before it can learn meaning.
Word-level gives short sequences and readable tokens, and it breaks on anything unseen. A vocabulary built from training text will not contain "unhappiness" if only "happiness" appeared, so the word becomes <UNK> and its meaning is lost entirely. Morphologically rich languages make this far worse.
Subword is the compromise everything modern uses. Common words stay whole; rare words split into meaningful fragments. Nothing is ever unknown, because the vocabulary includes the individual characters as a fallback.
How byte-pair encoding builds a vocabulary
BPE is the algorithm behind GPT's tokeniser, and it is simpler than its reputation.
Start with every character as a token.
Count all adjacent pairs in the training corpus.
Merge the most frequent pair into a new token.
Repeat until the vocabulary reaches the target size.
Run it on English text and the early merges are t+h → th, then th+e → the. Frequent sequences become single tokens; rare ones stay as fragments. The result is a vocabulary that spends its budget where the text actually is.
The consequence worth knowing: token count is not word count. English averages roughly 1.3 tokens per word. Code, unusual names, other scripts and emoji use far more — a Japanese sentence may cost three or four times as many tokens as its English translation, which is a real cost and latency difference when calling an API.
WordPiece (BERT) and SentencePiece (T5, Llama) are variants of the same idea. SentencePiece is notable for treating the input as a raw byte stream with no pre-tokenisation step, which means it needs no language-specific whitespace rules.
What tokenisation decides for you
Several model behaviours trace directly back to this step.
Why models are bad at spelling and character counting. "strawberry" may be three tokens, none of which is a letter. Asking how many r's it contains is asking the model to reason about something it cannot see.
Why numbers are handled inconsistently. "1234" might be one token or four, depending on the tokeniser, so arithmetic is learned over an inconsistent representation.
Why context limits are in tokens. A 128,000-token window is roughly 96,000 English words — and considerably fewer if the text is code or non-Latin script.
Why the tokeniser must ship with the model. Token id 1,547 means one thing to one tokeniser and something else to another. A mismatch produces fluent nonsense, not an error.
3. Experiment with the Visualization
The best way to grasp these concepts is to see them in action. Use the interactive tokenizer to build your intuition:
Compare Whitespace vs. Word Tokenization: Use the default sentence. First, run the "Whitespace" tokenizer and observe the output. Notice how "U.S.A." and "$50." are treated as single tokens. Now, switch to "Word (Regex)" and run it again. See how it intelligently separates the punctuation. This demonstrates the power of rule-based tokenization.
Analyze the Token and Vocab Counts: Type a sentence with repeated words, like "The quick brown fox jumps over the lazy dog." Notice that the "Token Count" is 9, but the "Unique" (vocabulary) count is 8, because "the" appears twice. This distinction is critical for building a model's vocabulary.
Test Edge Cases: Try tokenizing different kinds of text. Use a URL (like https://vizlearn.com), an email address (hello@example.com), or a sentence with hashtags (#NLP). How does each tokenizer handle these? Experimenting with these edge cases will show you the challenges and nuances of tokenization in real-world scenarios.
Advanced Tokenization: Sub-word Methods
Modern NLP models like BERT and GPT use more advanced techniques called sub-word tokenization (e.g., Byte-Pair Encoding or WordPiece). These methods break down rare words into smaller, known sub-word units. For example, "tokenization" might become ["token", "##ization"]. This allows the model to handle any word it encounters, even if it wasn't in the training data, preventing the "unknown token" problem and maintaining a manageable vocabulary size.
In code
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
tok.tokenize("Tokenisation isn't trivial.")
# ['token', '##isation', 'isn', "'", 't', 'trivial', '.']
ids = tok("Tokenisation isn't trivial.", return_tensors="pt")
ids["input_ids"] # includes [CLS] and [SEP] automatically
tok.decode(ids["input_ids"][0])
The ## prefix marks a continuation of the previous word, which is how WordPiece distinguishes "token" at the start of a word from the same letters inside one.
Special tokens matter and are easy to forget: [CLS] and [SEP] for BERT, <s> and </s> for others, plus [PAD] for batching and [MASK] for masked-language training. Using the tokeniser's __call__ adds them correctly; calling tokenize() alone does not.
For classical NLP the simpler tools still apply:
import re
re.findall(r"\b\w+\b", text.lower()) # crude word tokens
Splitting on whitespace looks adequate and is not: "don't", "state-of-the-art", "U.S.A." and "€19.99" all need decisions. NLTK's and spaCy's tokenisers encode those decisions; a regular expression encodes your assumptions.
Padding, truncation and attention masks
Models take fixed-size batches, so sequences of different lengths must be equalised.
Padding appends a filler token to short sequences. The attention mask is what tells the model to ignore those positions — without it, the model attends to padding as if it were content, and the results degrade for no visible reason.
Truncation cuts sequences that exceed the limit. Which end you cut matters: for classification the start of a document is often most informative, for question answering the relevant passage may be anywhere, and blind truncation silently discards the answer.
batch = tok(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
model(**batch) # attention_mask is included in batch
Padding to the longest sequence in the batch rather than to a fixed maximum — dynamic padding — is meaningfully faster when lengths vary.
Four tokenisers on the same sentence, including BPE from scratch
Character, word and subword tokenisation each trade vocabulary size against sequence length. Byte-pair encoding is trained here on a small corpus so you can watch the merges being learned.
example_01.pyNumPy
import re
from collections import Counter
text = "the unbelievable tokenizer tokenizes unbelievably fast"
print("input: %r" % text)
print()
chars = list(text)
print("1. CHARACTER level")
print(" tokens: %d, vocabulary: %d" % (len(chars), len(set(chars))))
print(" first 20: %s" % chars[:20])
print(" a tiny vocabulary and no unknown words ever -- at the cost of long")
print(" sequences, and attention cost grows with the SQUARE of that length.")
print()
words = text.split()
print("2. WORD level")
print(" tokens: %d, vocabulary: %d" % (len(words), len(set(words))))
print(" tokens: %s" % words)
print(" short sequences, but 'unbelievable' and 'unbelievably' are entirely")
print(" separate entries that share nothing. and any word not in the")
print(" vocabulary becomes <UNK>, which throws the information away.")
print()
print("3. SUBWORD -- the compromise everything modern uses.")
print(" train byte-pair encoding on a small corpus:")
corpus = ("low low low low low lower lower newest newest newest newest "
"newest newest widest widest widest lowest lowest")
vocab = Counter(corpus.split())
splits = {w: list(w) + ["</w>"] for w in vocab}
def pair_counts(splits, vocab):
c = Counter()
for w, freq in vocab.items():
sym = splits[w]
for i in range(len(sym) - 1):
c[(sym[i], sym[i + 1])] += freq
return c
merges = []
print("%6s %18s %8s %s" % ("step", "merged pair", "count", "example word after"))
for step in range(10):
counts = pair_counts(splits, vocab)
if not counts:
break
best, n = counts.most_common(1)[0]
merges.append(best)
for w in splits:
sym, out, i = splits[w], [], 0
while i < len(sym):
if i < len(sym) - 1 and (sym[i], sym[i + 1]) == best:
out.append(sym[i] + sym[i + 1]); i += 2
else:
out.append(sym[i]); i += 1
splits[w] = out
print("%6d %18s %8d %s"
% (step + 1, "%s + %s" % best, n, " ".join(splits["newest"])))
print()
print(" the final splits:")
for w in ("low", "lower", "newest", "widest", "lowest"):
print(" %-8s -> %s" % (w, " ".join(splits[w])))
print()
print(" 'est</w>' became a single token because it appeared often. that is")
print(" the whole algorithm: repeatedly merge the commonest adjacent pair.")
print(" nothing about suffixes was programmed in -- it fell out of counting.")
print()
def bpe_encode(word, merges):
sym = list(word) + ["</w>"]
for a, b in merges:
out, i = [], 0
while i < len(sym):
if i < len(sym) - 1 and sym[i] == a and sym[i + 1] == b:
out.append(a + b); i += 2
else:
out.append(sym[i]); i += 1
sym = out
return sym
print(" applied to words the tokeniser has never seen:")
for w in ("lowest", "newer", "wildest", "slowness"):
print(" %-10s -> %s" % (w, bpe_encode(w, merges)))
print(" nothing became <UNK>. an unknown word decomposes into pieces the")
print(" model does know, which is the property that makes subword")
print(" tokenisation the default.")
print()
print("THE TRADE. our toy tokeniser learned only %d merges from a corpus of"
% len(merges))
print("five words, so it can only be measured fairly on text from that")
print("corpus. a real BPE vocabulary has 32,000 merges learned from")
print("billions of words.")
print()
demo = "lowest newest widest lower low"
d_chars = list(demo)
d_words = demo.split()
d_sub = [t for w in d_words for t in bpe_encode(w, merges)]
print(" text: %r" % demo)
print("%-18s %10s %16s %s" % ("scheme", "tokens", "vocab needed", "unknown words"))
print("%-18s %10d %16d %s" % ("character", len(d_chars), len(set(d_chars)), "never"))
print("%-18s %10d %16s %s" % ("word", len(d_words), "~50,000+", "common"))
print("%-18s %10d %16s %s" % ("subword (BPE)", len(d_sub), "~32,000", "never"))
print(" %s" % d_sub)
print()
print(" subword sits between the two, which is the entire point of it.")
print()
print(" and here is what happens on text the tokeniser has never seen:")
away = "the unbelievable tokenizer"
print(" %r -> %d pieces"
% (away, len([t for w in away.split() for t in bpe_encode(w, merges)])))
print(" it falls back to characters, because none of those merges apply.")
print(" that is not a failure -- it still produces valid tokens with no")
print(" <UNK> anywhere. it is just what a 10-merge vocabulary can do.")
print()
print("things that bite in practice, with a real tokeniser:")
print(" a leading space is usually part of the token. 'cat' and ' cat' are")
print(" different tokens, which is why a prompt ending in a space can")
print(" behave oddly.")
print()
print(" numbers fragment. '2024' may be one, two or three tokens depending")
print(" on the vocabulary, and neighbouring numbers can split differently.")
print(" that inconsistency is part of why models are poor at arithmetic --")
print(" the digits are not reliably separate symbols.")
print()
print(" token count is not word count. English runs about 1.3 tokens per")
print(" word; code, non-Latin scripts and unusual names run considerably")
print(" higher. when you are billed per token that is a direct cost")
print(" difference, and when you have a context limit it is a direct")
print(" capacity difference.")
print()
print(" the vocabulary is frozen at training time. a tokeniser trained")
print(" before a word existed will always split it, however common it")
print(" later becomes.")
Output
Questions people ask
How many tokens is a word? About 1.3 for English prose, more for code, names and non-Latin scripts. Count with the actual tokeniser rather than estimating.
Should I lowercase before tokenising? Only if the tokeniser is uncased. Modern subword tokenisers usually handle case themselves, and lowercasing first destroys information they were trained to use.
Can I use a different tokeniser than the model was trained with? No. The ids would mean the wrong things.
Why does my model see <UNK> everywhere? A word-level vocabulary built from too little text. Use subword tokenisation.
Do I still need stemming and stop-word removal? For classical bag-of-words models, sometimes. For transformers, no — they use the full form and the function words carry syntax.
What is the vocabulary size trade-off? Larger means shorter sequences and a bigger embedding table; smaller means longer sequences and more fragmenting. 30,000–100,000 is the usual range.
Recap in one screen
Tokenisation turns text into integer ids; the choice of unit shapes everything after it.
Character level is robust and long; word level is short and breaks on unseen words; subword is the standard compromise.
BPE repeatedly merges the most frequent adjacent pair, so common words stay whole and rare ones fragment.
Token count is not word count, and it varies by language and content type.
Pad with an attention mask, truncate deliberately, and always ship the tokeniser with the model.
The key is representation quality: better representation usually improves downstream performance.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “What is Tokenization”?
Tokenization is the fundamental first step in any Natural Language Processing (NLP) pipeline. It's the process of breaking down a stream of raw text into smaller, meaningful units called tokens . These tokens can be words, characters, or sub-word units, depending on the chosen strategy.
What does this module say about “Why is Tokenization So Important”?
Imagine trying to understand a sentence by looking at it as one continuous string of letters. It would be nearly impossible. Tokenization provides the structure that machines need. By converting a sentence like "NLP is fascinating!" into tokens such as ["NLP", "is", "fascinating", "!"] , we create a list of items that a model can count, analyze, and assign meaning to.
What does this module say about “Exploring Different Tokenization Methods”?
There's no single "best" way to tokenize text; the right method depends on the task and the language. The interactive visualizer above lets you experiment with the most common strategies. Let's explore them:
Cheat sheet
NLP Tokenizer
Tokenization is the fundamental first step in any Natural Language Processing (NLP) pipeline. It's the process of breaking down a stream of raw text into smaller, meaningful units called tokens. These tokens can be words, characters, or sub-word units, depending on the chosen strategy. For a computer to understand human language, it must first dissect it into these discrete pieces, much like how we learn to read by first identifying individual words.
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.