Hide words in a sentence and let a bidirectional model guess them back. Click any token to mask it, then watch the model score candidates using context from both sides.
Input Sequence
CLICK TO MASK
← left contextboth directions feed the predictionright context →
Model Predictions
Click a token above to mask it
Overview
Why It Must Be Bidirectional
A causal model reads left to right and predicts what comes next. So it can never look ahead. MLM has no such restriction: the blank sits in the middle. And the model is free to use each token on both sides at once.
Mask soft in "the cat sat on the ___ mat" and the left context narrows it to a modifier while the right context (mat) rules out most adjectives. Neither side alone is enough. That is why MLM gives such strong grasp forms. And why it cannot be used to generate text. BERT masks 15% of tokens in each training case. But not all of them turn [MASK]: 80% are replaced with the mask token, 10% are replaced with a random word, and 10% are left unchanged. The random replacement prevents the model from learning that [MASK] is always present. And the unchanged tokens force it to build good forms even for visible words. The training objective is cross-entropy on only the masked spots. The model predicts over the full vocabulary at each masked slot. And the loss ignores the unmasked spots fully. After pretraining, the MLM head is discarded and the encoder is fine-tuned for downstream tasks (classification, NER, question answering) by attaching a task-set head. MLM cannot generate text because it has no notion of giving tokens left to right. It fills blanks, it does not continue sequences. GPT-style causal language modelling is the pretraining objective for generation. This page masks tokens in a sample sentence, shows the model's top predictions at each spot, and covers the 80/10/10 split and why it matters.
Hide words in a sentence and let a bidirectional model guess them back.
Masked Language Modeling: Learning by Filling Blanks
Masked Language Modeling (MLM) is the pretraining objective behind BERT and its descendants. The recipe is deceptively simple: take ordinary text, hide a fraction of the tokens, and train the model to reconstruct exactly what was hidden. No human labels required — the text supplies its own answer key.
Why It Must Be Bidirectional
A causal model reads left to right and predicts what comes next. So it can never look ahead. MLM has no such restriction: the blank sits in the middle. And the model is free to use each token on both sides at once.
Mask soft in "the cat sat on the ___ mat" and the left context narrows it to a modifier while the right context (mat) rules out most adjectives. Neither side alone is enough. That is why MLM gives such strong grasp forms. And why it cannot be used to generate text. BERT masks 15% of tokens in each training case. But not all of them turn [MASK]: 80% are replaced with the mask token, 10% are replaced with a random word, and 10% are left unchanged. The random replacement prevents the model from learning that [MASK] is always present. And the unchanged tokens force it to build good forms even for visible words. The training objective is cross-entropy on only the masked spots. The model predicts over the full vocabulary at each masked slot. And the loss ignores the unmasked spots fully. After pretraining, the MLM head is discarded and the encoder is fine-tuned for downstream tasks (classification, NER, question answering) by attaching a task-set head. MLM cannot generate text because it has no notion of giving tokens left to right. It fills blanks, it does not continue sequences. GPT-style causal language modelling is the pretraining objective for generation. This page masks tokens in a sample sentence, shows the model's top predictions at each spot, and covers the 80/10/10 split and why it matters.
The Curious 80 / 10 / 10 Split
BERT masks about 15% of tokens, but does not always insert a literal [MASK]. Of the chosen spots:
80% turn [MASK] — the standard fill-in-the-blank case.
10% are replaced by a random word. Forcing the model to notice when a token does not fit, rather than trusting any it sees.
10% are left unchanged — but still predicted, so the model keeps a useful form of each token, not just masked ones.
The reason is a train/test mismatch: [MASK] never appears during fine-tuning or inference. If each masked slot were a literal mask token, the model would learn features that only activate on a symbol it will never see again.
Masking Rate Is a Trade-off
Mask too little and each sentence teaches the model almost nothing. Training is slow because most spots carry no loss. Mask too much and you destroy the very context needed to make a prediction. Around 15% is the long-standing sweet spot, though later work has shown much higher rates can work with large enough models. Drag the rate slider and watch the sentence degrade. At 50% the left context often cannot pin the answer down.
Filling in the blanks
Masked language modelling hides some tokens and trains the model to recover them from both sides of the gap.
"The cat sat on the [MASK]" → predict "mat"
The gap from causal modelling is the way of data flow. Here the model sees all of it except the masked spots, so a token can be predicted from the words after it and before. That bidirectional context is what makes the coming forms strong for grasp tasks. And it is just why the model cannot generate, since there is no notion of "next".
BERT's set recipe masks 15% of tokens, and the composition is deliberate:
Treatment
Share of masked tokens
Replaced with [MASK]
80%
Replaced with a random token
10%
Left unchanged
10%
The last two look odd and solve a real problem. [MASK] never appears at fine-tuning or inference time, so a model trained only on it would be mismatched with how it is used. Including corrupted and unchanged tokens forces the model to build a good form of each spot, not just the masked ones.
Why 15%, and the speed cost
Too little masking and each pass teaches almost nothing. Too much and there is not enough context left to make prediction doable.
15% became the convention, and later work suggests the optimum is task- and scale-dependent. Some studies find 40% works better for larger models.
There is a real speed result either way:
Masked LM
Causal LM
Predictions per forward pass
15% of tokens
100% of tokens
Sees
Both directions
Past only
Can generate
No
Yes
A causal model extracts a training sign from each spot in one pass; a masked model from roughly one in seven. That is a large gap in training speed. And it is one of a few reasons the field shifted towards causal modelling for large-scale pretraining.
ELECTRA addressed it right with a different objective — a small generator replaces tokens. And the main model classifies each spot as first or replaced. This gives a sign at all spots.
What it gives, and what it is for
The output of masked pretraining is a model whose hidden states are excellent context forms. That is the product; the mask-filling task was only the means.
Those forms are what get used:
Embeddings for search, likeness and clustering — especially after contrastive fine-tuning, as in Sentence-BERT.
Token classification — named entities, part of speech, where bidirectional context truly helps.
Sentence classification — sentiment, intent, topic, via the [CLS] spot or pooled states.
Span extraction — extractive question answering, predicting start and end spots.
For all of these, an encoder fine-tuned on a few thousand labelled cases is often as exact as prompting a much larger model, and orders of size cheaper to run. A DistilBERT classifier answers in milliseconds on a CPU.
Exploration guide
Mask a single strongly-constrained token such as mat. The top candidate should dominate — the nearby words leave little freedom.
Now mask an adjective like soft. Confidence spreads across a few plausible words: the context permits a whole family of answers.
Load the "bank / river" preset and mask bank. The words after it are what disambiguate the sense. Just the data a left-to-right model would not yet have.
Mask two adjacent tokens. Precision drops sharply, because MLM predicts each blank on its own and cannot coordinate between them. A genuine known limitation of the objective.
About the model in this lab
Predictions come from a small context-scoring model built live in your browser from the demo corpus shown in the vocabulary count. It ranks candidate words by how often they co-occur with the surviving context. It is a genuine math, not a canned list. But it is far simpler than a transformer. Treat the shape of the results as the lesson, not the exact percentages.
In one line
MLM turns any raw text into supervised training data by deleting parts of it. Because the blank is surrounded rather than trailing, the model learns deep bidirectional grasp. Ideal for classification, retrieval and question answering, and unsuitable for generation. That generative job belongs to causal models, which predict strictly forward.
The variants that improved on BERT
Model
What changed
RoBERTa
More data, longer training, dynamic masking, no next-sentence prediction
ALBERT
Parameter sharing across layers, factorised embeddings
DeBERTa
Disentangled attention for content and position
ELECTRA
Replaced-token detection instead of masking
DistilBERT
Distilled from BERT: 40% smaller, 60% faster, ~97% of the quality
RoBERTa is the instructive one. Same design, same objective, and much better results. From training longer on more data with larger batches, masking dynamically (a different mask each epoch rather than one fixed at preprocessing), and dropping BERT's next-sentence-prediction objective. This turned out to be unhelpful.
That is a useful general lesson: BERT's first recipe was not best. And a large part of the later improvement came from training procedure rather than design.
Next-sentence prediction asked whether two segments were adjacent in the first text. It was intended to teach discourse links and mostly taught topic matching. This the masked objective now provided.
Masked versus causal, in practice
The split looked permanent for a few years — encoders for grasp, decoders for generation. What changed is scale: a large decoder, prompted rightly, does classification and extraction well enough that the bidirectional edge stopped being decisive for most uses.
So current practice is roughly:
Need
Reach for
Embeddings for search or clustering
A masked-pretrained encoder, contrastively fine-tuned
Cheap high-volume classification
A fine-tuned small encoder
Token-level labelling
An encoder
Generation of any kind
A decoder
Zero-shot or few-shot on an unknown task
A decoder
Lowest possible inference cost
An encoder
The rows favouring encoders share a shape: a fixed task with labelled data and a cost or latency constraint. That describes a great many production systems, which is why BERT-family models remain very widely deployed despite receiving far less attention.
Real notes
from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")
text = "The capital of France is [MASK]."
ids = tok(text, return_tensors="pt")
logits = model(**ids).logits
pos = (ids.input_ids[0] == tok.mask_token_id).nonzero(as_tuple=True)[0]
top = logits[0, pos].topk(5).indices[0]
print(tok.convert_ids_to_tokens(top)) # ['paris', 'lyon', 'marseille', ...]
Three things worth knowing when fine-tuning one:
Continued pretraining helps on specialised domains. Running the masked objective on your own unlabelled corpus before supervised fine-tuning often gives a measurable gain, and needs no labels.
The [CLS] form is not a good sentence embedding out of the box. It was not trained for likeness. Use a model fine-tuned contrastively, or masked mean pooling.
512 tokens is the usual limit. Longer documents need chunking, or a long-context variant such as Longformer.
Why 15 percent, and what the 80/10/10 split is really for
Masked language modelling has two numbers in it that look arbitrary. A 15 percent masking rate and an 80/10/10 split of what to do with each masked token. Both are answers to set problems, and both are visible in the arithmetic.
example_01.pyNumPy
import numpy as np
SENT = ["the", "cat", "sat", "on", "the", "soft", "mat", "by", "the", "fire",
"and", "slept", "for", "an", "hour", "or", "so", "quietly", "all",
"afternoon"]
T = len(SENT)
print("A SENTENCE OF %d TOKENS. masked language modelling hides some and" % T)
print("asks the model to restore them, using BOTH sides as context:")
print(" %s" % " ".join(SENT))
print()
print("THE MASKING RATE IS A TRADE-OFF between two things that pull in")
print("opposite directions:")
print("%-10s %14s %20s %22s"
% ("rate", "tokens masked", "training signal", "context left intact"))
for r in (0.05, 0.15, 0.30, 0.50, 0.80):
m = int(round(T * r))
print("%-10s %14d %20s %20.0f%%"
% ("%.0f%%" % (100 * r), m,
"%d prediction%s" % (m, "" if m == 1 else "s"), 100 * (1 - r)))
print(" more masking means more predictions per sentence, so more")
print(" learning signal per forward pass. it also means less context")
print(" left to predict FROM.")
print(" at 80 percent the model is being asked to reconstruct a sentence")
print(" from almost nothing, which is not a language task, it is a")
print(" guessing game. 15 percent is where the two curves cross in")
print(" practice, and it is empirical rather than derived.")
print()
Output
Building on the same setup, the next block shows the left cases.
example_02.pyNumPy
import numpy as np
SENT = ["the", "cat", "sat", "on", "the", "soft", "mat", "by", "the", "fire",
"and", "slept", "for", "an", "hour", "or", "so", "quietly", "all",
"afternoon"]
T = len(SENT)
print("AND THE EFFICIENCY COST NOBODY MENTIONS. compare with causal")
print("language modelling, which gets a prediction at EVERY position:")
print("%-34s %16s %20s"
% ("objective", "predictions", "per 1000 tokens"))
print("%-34s %16s %20d"
% ("causal LM", "%d of %d" % (T - 1, T), int(1000 * (T - 1) / T)))
for r in (0.15, 0.30):
print("%-34s %16s %20d"
% ("masked LM at %.0f%%" % (100 * r),
"%d of %d" % (int(round(T * r)), T), int(1000 * r)))
print(" a causal model learns from %d of every %d tokens; a masked model"
% (T - 1, T))
print(" at 15 percent learns from %d. that is roughly %.1fx more"
% (int(round(T * 0.15)), (T - 1) / float(T) / 0.15))
print(" training signal per token of data, and it is the main reason the")
print(" large models everyone uses are causal rather than masked -- not")
print(" because bidirectionality is bad, but because it is expensive.")
print()
Output
The below block covers the left steps.
example_03.pyNumPy
import numpy as np
SENT = ["the", "cat", "sat", "on", "the", "soft", "mat", "by", "the", "fire",
"and", "slept", "for", "an", "hour", "or", "so", "quietly", "all",
"afternoon"]
T = len(SENT)
rng = np.random.default_rng(3)
print("THE 80/10/10 SPLIT. of the tokens chosen for masking:")
print("%-30s %10s %s" % ("what happens to it", "share", "why"))
print("%-30s %10s %s" % ("replaced with [MASK]", "80%", "the actual task"))
print("%-30s %10s %s" % ("replaced with a RANDOM token", "10%", "see below"))
print("%-30s %10s %s" % ("left unchanged", "10%", "see below"))
print()
print(" the last two are the interesting ones, and both exist to fix a")
print(" MISMATCH between training and use:")
print(" during training, the model sees [MASK] tokens.")
print(" during use, it never does -- [MASK] appears in no real text.")
print(" if 100 percent of masked positions were [MASK], the model could")
print(" learn a shortcut: 'only bother building a real representation")
print(" where I see [MASK], and copy the input everywhere else'. that")
print(" model would score well on the training objective and be useless")
print(" as a feature extractor, which is the only thing it is for.")
print()
print(" the 10 percent RANDOM replacement is what forbids the shortcut:")
chosen = [1, 5, 9]
print("%-8s %-12s %-16s %s" % ("position", "original", "model sees", "must predict"))
for i, (pos, action) in enumerate(zip(chosen, ["[MASK]", "random", "unchanged"])):
seen = {"[MASK]": "[MASK]", "random": "banana",
"unchanged": SENT[pos]}[action]
print("%-8d %-12s %-16s %s" % (pos, SENT[pos], seen, SENT[pos]))
print(" the model cannot tell, from the input alone, whether the token")
print(" in front of it is correct. so it has to build a real")
print(" representation of EVERY position, on the chance that this one is")
print(" the corrupted one.")
print(" the 10 percent unchanged does the same job from the other side:")
print(" sometimes the answer is 'this token is already right', so")
print(" 'predict something different' is not a safe default either.")
print()
Output
The next block goes on with the left demonstrations.
example_04.pyNumPy
import numpy as np
SENT = ["the", "cat", "sat", "on", "the", "soft", "mat", "by", "the", "fire",
"and", "slept", "for", "an", "hour", "or", "so", "quietly", "all",
"afternoon"]
T = len(SENT)
print("WHAT THE BIDIRECTIONALITY BUYS. a token's representation can use")
print("both sides, which a causal model cannot do at all:")
print(" the ___ sat on the mat <- left context only")
print(" the ___ sat on the mat, purring and licking its paws")
print(" a causal model predicting position 1 sees only 'the'. a masked")
print(" model sees the entire rest of the sentence, including 'purring',")
print(" which is what actually identifies the animal.")
print(" that is why masked models are better at classification and")
print(" retrieval encoding, and cannot generate text at all: generation")
print(" needs the guarantee that position i never saw i+1, and masked")
print(" language modelling is built on breaking exactly that guarantee.")
print()
print("SO THE TWO OBJECTIVES ARE FOR DIFFERENT JOBS:")
print("%-26s %-22s %s" % ("", "causal LM", "masked LM"))
for row in (("sees", "the left only", "both sides"),
("predictions per token", "1.0", "0.15"),
("can generate", "yes", "no"),
("good for", "generation, chat", "embeddings, classification"),
("typical example", "GPT, Llama", "BERT, and every")):
print("%-26s %-22s %s" % row)
print(" and the last row explains something that looks odd from a")
print(" distance: almost every EMBEDDING model in a RAG pipeline is a")
print(" masked model, while the thing that writes the answer is a causal")
print(" one. a retrieval system usually runs both, for the reasons in")
print(" the two middle rows.")
Output
Questions people ask
Why can BERT not generate text? It has no notion of "next". Its objective assumes nearby context exists. Iterative mask-filling works poorly.
Is masked language modelling obsolete? Not for embeddings and cheap classification, where encoders remain the real choice. For general-purpose models, causal pretraining rules.
Why the 80/10/10 split? Because [MASK] does not exist at fine-tuning time; the random and unchanged cases prevent the model relying on the token's presence.
Should I continue pretraining on my domain? If you have a large unlabelled domain corpus, often yes. It is cheap and often helps.
What is [CLS] for? A spot whose final form is used as a sequence summary for classification.
Can I use a decoder for embeddings? Yes, by pooling hidden states, and purpose-trained embedding models often do better. This is because they are trained with a likeness objective.
Recap in one screen
Hide about 15% of tokens and predict them from both ways; the forms are the real product.
The 80/10/10 masking split exists because [MASK] never appears at fine-tuning time.
Bidirectional context is why encoders excel at grasp and cannot generate.
Causal modelling trains on each spot per pass, which is a large speed edge at scale.
Encoders remain the right tool for embeddings, token labelling and cheap high-volume classification.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
What does this module say about “Why It Must Be Bidirectional”?
What does this module say about “The Curious 80 / 10 / 10 Split”?
Cheat sheet
What is Masked Language Modeling?
Hide 15 percent of tokens and train the model to reconstruct them using both sides of the sentence — BERT's pretraining objective.
GEN AI · vizlearn.in/gen_ai/masked_language_modeling.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.