vizlearn

Home / Natural Language Processing

Text Tokenization

Natural Language ProcessingBy Ashish JangraUpdated 30 September 2026

Break down raw text into meaningful units (tokens) using various segmentation strategies.

Input Corpus

Tokenization Method

Whitespace

Split by spaces only

Word (Regex)

Split words & punctuation

Sentence

Split by delimiters (. ! ?)

Character

Split every character

Token Stream

0 Tokens 0 Unique

Click 'Tokenize' to break the text apart.

Overview

Break down raw text into meaningful units (tokens) using various segmentation strategies.

In this guide

What is Tokenization?
1. Why is Tokenization So Key?
2. Exploring Else Tokenization Ways
Cutting text into pieces a net can count
How byte-pair encoding builds a vocabulary
What tokenisation decides for you
3. Test with the Visualization
Advanced Tokenization: Sub-word Ways
In code
Padding, truncation and attention masks
Four tokenisers on the same sentence, with BPE from scratch
Questions people ask
Recap in one screen

A Beginner's Guide to Text Tokenization

Learn how to break down text into meaningful units for NLP models.

What is Tokenization?

Tokenization is the basic first step in any Natural Language Handling (NLP) pipeline. It's the process of breaking down a stream of raw text into less, meaningful units called tokens. These tokens can be words, characters, or sub-word units. It depends on the chosen plan. For a computer to understand human language, it must first dissect it into these discrete pieces, much like how we learn to read by first finding single words. Three plans dominate. Word-level tokenization splits on whitespace and punctuation, giving familiar units but a vocabulary that can run into the millions. And any word missing from it turns an unknown token. Character-level tokenization uses a tiny vocabulary (the alphabet) but gives quite long sequences that the net must reassemble into meaning. Subword tokenization (BPE, WordPiece, SentencePiece) splits rare words into frequent pieces while keeping common words whole, balancing vocabulary size against sequence length. GPT uses BPE; BERT uses WordPiece. The choice of tokenizer decides the net's vocabulary, its max sequence length, and how it handles unseen words — making it one of the most consequential decisions in an NLP pipeline, still though it receives far less attention than the net design. This page tokenizes a sample sentence with all three plans, checks the coming token counts and vocabulary sizes, and shows why subword ways have turn the default.

1. Why is Tokenization So Key?

Imagine trying to understand a sentence by looking at it as one smooth string of letters. It would be nearly not doable. Tokenization gives the structure that machines need. By turning a sentence like "NLP is fascinating!" into tokens such as ["NLP", "is", "fascinating", "!"], we create a list of things that a net can count, study, and give meaning to. This process is the gateway to almost each NLP task, with sentiment review, machine translation, and text summarization.

2. Exploring Else Tokenization Ways

There's no single "best" way to tokenize text. The right way depends on the task and the language. The interactive visualizer up lets you test with the most common plans. This page covers them:

Whitespace Tokenization: This is the simplest way. It splits the text based on empty spaces. While fast and straightforward, it's much too basic. For case, it would treat "New York" as two separate tokens, ["New", "York"], and might fail to separate a word from punctuation, like in "fun!".
Word (Regex) Tokenization: A more complex approach that uses regular terms (regex) to define what constitutes a word. This way is much better at handling punctuation and special cases. For instance, it can correctly separate "fun!" into ["fun", "!"] and can be configured to handle hyphenated words or contractions like "don't".
Sentence Tokenization: Instead of words, this way splits the text into single sentences. It finds sentence edges with delimiters like periods (.), question marks (?), and exclamation marks (!). This is a key preprocessing step for tasks that need to understand the context of a full sentence, such as document summarization.
Character Tokenization: This way breaks the text down to its most granular level: single characters. The sentence "Hello" turns ["H", "e", "l", "l", "o"]. This approach is useful for languages without clear word edges or for tasks like spell-checking and text generation, as it creates a quite small, manageable vocabulary.

Cutting text into pieces a net can count

A net cannot consume a string. It consumes integers, each pointing at a row of an embedding table. Tokenisation is the step that decides what those integers stand for. And the choice shapes all of it downstream.

Three levels are doable:

Level	"unhappiness" becomes	Vocabulary size
Character	u, n, h, a, p, ...	~100
Word	unhappiness	100,000+
Subword	un, happi, ness	30,000–100,000

Character-level needs a tiny vocabulary and never meets an unknown symbol, but sequences turn quite long and the net has to learn spelling before it can learn meaning.

Word-level gives short sequences and readable tokens, and it breaks on a thing unseen. A vocabulary built from training text will not hold "unhappiness" if just "happiness" appeared. So the word turns <UNK> and its meaning is lost fully. Morphologically rich languages make this far worse.

Subword is the compromise all of it modern uses. Common words stay whole; rare words split into meaningful fragments. Nothing is ever unknown, since the vocabulary includes the single characters as a fallback.

How byte-pair encoding builds a vocabulary

BPE is the algorithm behind GPT's tokeniser, and it is simpler than its reputation.

Start with each character as a token.
Count all next pairs in the training corpus.
Merge the most frequent pair into a new token.
Repeat until the vocabulary reaches the target size.

Run it on English text and the early merges are t+h → th. Then th+e → the. Frequent sequences turn single tokens; rare ones stay as fragments. The yield is a vocabulary that spends its budget where the text really is.

The yield worth knowing: token count is not word count. English averages roughly 1.3 tokens per word. Code, unusual names, else scripts and emoji use far more. A Japanese sentence may cost three or four times as many tokens as its English translation. This is a real cost and latency gap when calling an API.

WordPiece (BERT) and SentencePiece (T5, Llama) are variants of the same idea. SentencePiece is notable for treating the input as a raw byte stream with no pre-tokenisation step. This means it needs no language-set whitespace rules.

What tokenisation decides for you

A few net behaviours trace right back to this step.

Why nets are bad at spelling and character counting. "strawberry" may be three tokens, none of which is a letter. Asking how many r's it holds is asking the net to reason about a thing it cannot see.

Why counts are handled inconsistently. "1234" might be one token or four. It depends on the tokeniser. So arithmetic is learned over an inconsistent form.

Why context limits are in tokens. A 128,000-token window is roughly 96,000 English words. And much fewer if the text is code or non-Latin script.

Why the tokeniser must ship with the net. Token id 1,547 means one thing to one tokeniser and a thing else to one more. A mismatch gives fluent nonsense, not an error.

3. Test with the Visualization

The best way to grasp these concepts is to see them in action. Use the interactive tokenizer to build your intuition:

Check Whitespace vs. Word Tokenization: Use the default sentence. First, run the "Whitespace" tokenizer and observe the output. Notice how "U.S.A." and "$50." are treated as single tokens. Now, switch to "Word (Regex)" and run it again. See how it intelligently separates the punctuation. This shows what rule-based tokenization.
Study the Token and Vocab Counts: Type a sentence with repeated words, like "The quick brown fox jumps over the lazy dog." Notice that the "Token Count" is 9. But the "Unique" (vocabulary) count is 8, since "the" appears twice. This distinction is critical for building a net's vocabulary.
Test Edge Cases: Try tokenizing else kinds of text. Use a URL (like https://vizlearn.com), an email address (hello@case.com), or a sentence with hashtags (#NLP). How does each tokenizer handle these? Experimenting with these edge cases will show you the challenges and nuances of tokenization in real-world scenarios.

Advanced Tokenization: Sub-word Ways

Modern NLP nets like BERT and GPT use more advanced ways called sub-word tokenization (e.g., Byte-Pair Encoding or WordPiece). These ways break down rare words into less, known sub-word units. For case, "tokenization" might turn ["token", "##ization"]. This lets the net to handle any word it hits, still if it wasn't in the training data. This stops the "unknown token" problem and maintaining a manageable vocabulary size.

In code

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("bert-base-uncased")

tok.tokenize("Tokenisation isn't trivial.")
# ['token', '##isation', 'isn', "'", 't', 'trivial', '.']

ids = tok("Tokenisation isn't trivial.", return_tensors="pt")
ids["input_ids"]          # includes [CLS] and [SEP] automatically
tok.decode(ids["input_ids"][0])

The ## prefix marks a follow-up of the last word, which is how WordPiece distinguishes "token" at the start of a word from the same letters inside one.

Special tokens matter and are easy to forget: [CLS] and [SEP] for BERT, <s> and </s> for rest, plus [PAD] for batching and [MASK] for masked-language training. With the tokeniser's __call__ adds them correctly. Calling tokenize() alone does not.

For classical NLP the simpler tools still apply:

import re
re.findall(r"\b\w+\b", text.lower())     # crude word tokens

Splitting on whitespace looks adequate and is not: "don't", "state-of-the-art", "U.S.A." and "€19.99" all need decisions. NLTK's and spaCy's tokenisers encode those decisions. A regular term encodes your assumptions.

Padding, truncation and attention masks

Nets take fixed-size batches, so sequences of else lengths must be equalised.

Padding appends a filler token to short sequences. The attention mask is what tells the net to ignore those spots. Without it, the net attends to padding as if it were content. And the yields degrade for no visible reason.

Truncation cuts sequences that exceed the limit. Which end you cut matters: for classification the start of a document is much most informative, for question answering the apt passage may be anywhere, and blind truncation silently discards the answer.

batch = tok(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
model(**batch)          # attention_mask is included in batch

Padding to the longest sequence in the batch rather than to a fixed max — dynamic padding — is meaningfully faster when lengths vary.

Four tokenisers on the same sentence, with BPE from scratch

Character, word and subword tokenisation each trade vocabulary size against sequence length. Byte-pair encoding is trained here on a small corpus so you can watch the merges being learned.

example_01.pyNumPy

import re
from collections import Counter

text = "the unbelievable tokenizer tokenizes unbelievably fast"
print("input: %r" % text)
print()

chars = list(text)
print("1. CHARACTER level")
print("   tokens: %d, vocabulary: %d" % (len(chars), len(set(chars))))
print("   first 20: %s" % chars[:20])
print("   a tiny vocabulary and no unknown words ever -- at the cost of long")
print("   sequences, and attention cost grows with the SQUARE of that length.")
print()

words = text.split()
print("2. WORD level")
print("   tokens: %d, vocabulary: %d" % (len(words), len(set(words))))
print("   tokens: %s" % words)
print("   short sequences, but 'unbelievable' and 'unbelievably' are entirely")
print("   separate entries that share nothing. and any word not in the")
print("   vocabulary becomes <UNK>, which throws the information away.")
print()

print("3. SUBWORD -- the compromise everything modern uses.")
print("   train byte-pair encoding on a small corpus:")
corpus = ("low low low low low lower lower newest newest newest newest "
          "newest newest widest widest widest lowest lowest")
vocab = Counter(corpus.split())
splits = {w: list(w) + ["</w>"] for w in vocab}

Output

Building on the same setup, the next block shows the left cases.

example_02.pyNumPy

import re
from collections import Counter

def pair_counts(splits, vocab):
    c = Counter()
    for w, freq in vocab.items():
        sym = splits[w]
        for i in range(len(sym) - 1):
            c[(sym[i], sym[i + 1])] += freq
    return c
print("done")

Output

The down block covers the left steps.

example_03.pyNumPy

import re
from collections import Counter

merges = []
print("%6s %18s %8s %s" % ("step", "merged pair", "count", "example word after"))
for step in range(10):
    counts = pair_counts(splits, vocab)
    if not counts:
        break
    best, n = counts.most_common(1)[0]
    merges.append(best)
    for w in splits:
        sym, out, i = splits[w], [], 0
        while i < len(sym):
            if i < len(sym) - 1 and (sym[i], sym[i + 1]) == best:
                out.append(sym[i] + sym[i + 1]); i += 2
            else:
                out.append(sym[i]); i += 1
        splits[w] = out
    print("%6d %18s %8d %s"
          % (step + 1, "%s + %s" % best, n, " ".join(splits["newest"])))
print()
print("   the final splits:")
for w in ("low", "lower", "newest", "widest", "lowest"):
    print("      %-8s -> %s" % (w, " ".join(splits[w])))
print()
print("   'est</w>' became a single token because it appeared often. that is")
print("   the whole algorithm: repeatedly merge the commonest adjacent pair.")
print("   nothing about suffixes was programmed in -- it fell out of counting.")
print()

Output

The next block goes on with the left demonstrations.

example_04.pyNumPy

import re
from collections import Counter

def bpe_encode(word, merges):
    sym = list(word) + ["</w>"]
    for a, b in merges:
        out, i = [], 0
        while i < len(sym):
            if i < len(sym) - 1 and sym[i] == a and sym[i + 1] == b:
                out.append(a + b); i += 2
            else:
                out.append(sym[i]); i += 1
        sym = out
    return sym

print("   applied to words the tokeniser has never seen:")
for w in ("lowest", "newer", "wildest", "slowness"):
    print("      %-10s -> %s" % (w, bpe_encode(w, merges)))
print("   nothing became <UNK>. an unknown word decomposes into pieces the")
print("   model does know, which is the property that makes subword")
print("   tokenisation the default.")
print()

Output

Building on the same setup, the next block shows the left cases.

example_05.pyNumPy

import re
from collections import Counter

print("THE TRADE. our toy tokeniser learned only %d merges from a corpus of"
      % len(merges))
print("five words, so it can only be measured fairly on text from that")
print("corpus. a real BPE vocabulary has 32,000 merges learned from")
print("billions of words.")
print()
demo = "lowest newest widest lower low"
d_chars = list(demo)
d_words = demo.split()
d_sub = [t for w in d_words for t in bpe_encode(w, merges)]
print("   text: %r" % demo)
print("%-18s %10s %16s %s" % ("scheme", "tokens", "vocab needed", "unknown words"))
print("%-18s %10d %16d %s" % ("character", len(d_chars), len(set(d_chars)), "never"))
print("%-18s %10d %16s %s" % ("word", len(d_words), "~50,000+", "common"))
print("%-18s %10d %16s %s" % ("subword (BPE)", len(d_sub), "~32,000", "never"))
print("      %s" % d_sub)
print()
print("   subword sits between the two, which is the entire point of it.")
print()
print("   and here is what happens on text the tokeniser has never seen:")
away = "the unbelievable tokenizer"
print("      %r -> %d pieces"

Output

The down block covers the left steps.

example_06.pyNumPy

      % (away, len([t for w in away.split() for t in bpe_encode(w, merges)])))
print("      it falls back to characters, because none of those merges apply.")
print("      that is not a failure -- it still produces valid tokens with no")
print("      <UNK> anywhere. it is just what a 10-merge vocabulary can do.")
print()

print("things that bite in practice, with a real tokeniser:")
print("   a leading space is usually part of the token. 'cat' and ' cat' are")
print("   different tokens, which is why a prompt ending in a space can")
print("   behave oddly.")
print()
print("   numbers fragment. '2024' may be one, two or three tokens depending")
print("   on the vocabulary, and neighbouring numbers can split differently.")
print("   that inconsistency is part of why models are poor at arithmetic --")
print("   the digits are not reliably separate symbols.")
print()
print("   token count is not word count. English runs about 1.3 tokens per")
print("   word; code, non-Latin scripts and unusual names run considerably")
print("   higher. when you are billed per token that is a direct cost")
print("   difference, and when you have a context limit it is a direct")
print("   capacity difference.")
print()
print("   the vocabulary is frozen at training time. a tokeniser trained")
print("   before a word existed will always split it, however common it")
print("   later becomes.")

Output

Questions people ask

How many tokens is a word? About 1.3 for English prose, more for code, names and non-Latin scripts. Count with the actual tokeniser rather than estimating.

Should I lowercase before tokenising? Just if the tokeniser is uncased. Modern subword tokenisers much handle case them, and lowercasing first destroys data they were trained to use.

Can I use a else tokeniser than the net was trained with? No. The ids would mean the wrong things.

Why does my net see <UNK> all over? A word-level vocabulary built from too little text. Use subword tokenisation.

Do I still need stemming and stop-word cutting? For classical bag-of-words nets, sometimes. For transformers, no — they use the full form and the task words carry syntax.

What is the vocabulary size trade-off? More means shorter sequences and a bigger embedding table. Less means longer sequences and more fragmenting. 30,000–100,000 is the usual range.

Recap in one screen

Tokenisation turns text into integer ids. The choice of unit shapes all of it past it.
Character level is reliable and long. Word level is short and breaks on unseen words; subword is the standard compromise.
BPE repeatedly merges the most frequent next pair, so common words stay whole and rare ones fragment.
Token count is not word count, and it varies by language and content type.
Pad with an attention mask, truncate on purpose, and always ship the tokeniser with the net.

The key is representation quality: better representation usually improves downstream performance.

Recall check

0 of 3

Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.

What does this module say about “What is Tokenization”?
What does this module say about “Why is Tokenization So Key”?
What does this module say about “Exploring Else Tokenization Ways”?

Cheat sheet

NLP Tokenizer

Split text into words, characters or sentences and see how the tokenizer you choose changes what a model actually reads.

NLP · vizlearn.in/natural_language_processing/tokenization.html

About the author

Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.

About VizLearn GitHub LinkedIn Contact

Text Tokenization

Input Corpus

Tokenization Method

Whitespace

Word (Regex)

Sentence

Character

Token Stream

Overview

What is Tokenization?

A Beginner's Guide to Text Tokenization

What is Tokenization?

1. Why is Tokenization So Key?

2. Exploring Else Tokenization Ways

Cutting text into pieces a net can count

How byte-pair encoding builds a vocabulary

What tokenisation decides for you

3. Test with the Visualization

Advanced Tokenization: Sub-word Ways

In code

Padding, truncation and attention masks

Four tokenisers on the same sentence, with BPE from scratch

Questions people ask

Recap in one screen

Recall check

NLP Tokenizer

More in NLP

About the author