Hide words in a sentence and let a bidirectional model guess them back. Click any token to mask it, then watch the model score candidates using context from both sides.
Overview
Why It Must Be Bidirectional
A causal model reads left to right and predicts what comes next, so it can never look ahead. MLM has no such restriction: the blank sits in the middle, and the model is free to use every token on both sides at once.
Mask soft in "the cat sat on the ___ mat" and the left context narrows it to a modifier while the right context (mat) rules out most adjectives. Neither side alone is enough. That is why MLM produces such strong understanding representations — and why it cannot be used to generate text.
Input Sequence
CLICK TO MASK
← left contextboth directions feed the predictionright context →
Model Predictions
Click a token above to mask it
Masked Language Modeling: Learning by Filling Blanks
Masked Language Modeling (MLM) is the pretraining objective behind BERT and its descendants. The recipe is deceptively simple: take ordinary text, hide a fraction of the tokens, and train the model to reconstruct exactly what was hidden. No human labels required — the text supplies its own answer key.
The Curious 80 / 10 / 10 Split
BERT masks about 15% of tokens, but does not always insert a literal [MASK]. Of the chosen positions:
80% become [MASK] — the standard fill-in-the-blank case.
10% are replaced by a random word — forcing the model to notice when a token does not fit, rather than trusting whatever it sees.
10% are left unchanged — but still predicted, so the model keeps a useful representation of every token, not just masked ones.
The reason is a train/test mismatch: [MASK] never appears during fine-tuning or inference. If every masked slot were a literal mask token, the model would learn features that only activate on a symbol it will never see again.
Masking Rate Is a Trade-off
Mask too little and each sentence teaches the model almost nothing — training is slow because most positions carry no loss. Mask too much and you destroy the very context needed to make a prediction. Around 15% is the long-standing sweet spot, though later work has shown much higher rates can work with large enough models. Drag the rate slider and watch the sentence degrade: at 50% the remaining context often cannot pin the answer down.
Filling in the blanks
Masked language modelling hides some tokens and trains the model to recover them from both sides of the gap.
"The cat sat on the [MASK]" → predict "mat"
The difference from causal modelling is the direction of information flow. Here the model sees everything except the masked positions, so a token can be predicted from the words after it as well as before. That bidirectional context is what makes the resulting representations strong for understanding tasks — and it is exactly why the model cannot generate, since there is no notion of "next".
BERT's specific recipe masks 15% of tokens, and the composition is deliberate:
Treatment
Share of masked tokens
Replaced with [MASK]
80%
Replaced with a random token
10%
Left unchanged
10%
The last two look odd and solve a real problem. [MASK] never appears at fine-tuning or inference time, so a model trained only on it would be mismatched with how it is used. Including corrupted and unchanged tokens forces the model to build a good representation of every position, not just the masked ones.
Why 15%, and the efficiency cost
Too little masking and each pass teaches almost nothing. Too much and there is not enough context left to make prediction possible.
15% became the convention, and later work suggests the optimum is task- and scale-dependent — some studies find 40% works better for larger models.
There is a real efficiency consequence either way:
Masked LM
Causal LM
Predictions per forward pass
15% of tokens
100% of tokens
Sees
Both directions
Past only
Can generate
No
Yes
A causal model extracts a training signal from every position in one pass; a masked model from roughly one in seven. That is a substantial difference in training efficiency, and it is one of several reasons the field shifted towards causal modelling for large-scale pretraining.
ELECTRA addressed it directly with a different objective — a small generator replaces tokens, and the main model classifies every position as original or replaced, giving a signal at all positions.
What it produces, and what it is for
The output of masked pretraining is a model whose hidden states are excellent contextual representations. That is the product; the mask-filling task was only the means.
Those representations are what get used:
Embeddings for search, similarity and clustering — particularly after contrastive fine-tuning, as in Sentence-BERT.
Token classification — named entities, part of speech, where bidirectional context genuinely helps.
Sentence classification — sentiment, intent, topic, via the [CLS] position or pooled states.
Span extraction — extractive question answering, predicting start and end positions.
For all of these, an encoder fine-tuned on a few thousand labelled examples is often as accurate as prompting a much larger model, and orders of magnitude cheaper to run. A DistilBERT classifier answers in milliseconds on a CPU.
Exploration guide
Mask a single strongly-constrained token such as mat. The top candidate should dominate — the surrounding words leave little freedom.
Now mask an adjective like soft. Confidence spreads across several plausible words: the context permits a whole family of answers.
Load the "bank / river" preset and mask bank. The words after it are what disambiguate the sense — exactly the information a left-to-right model would not yet have.
Mask two adjacent tokens. Accuracy drops sharply, because MLM predicts each blank independently and cannot coordinate between them — a genuine known limitation of the objective.
About the model in this lab
Predictions come from a small context-scoring model built live in your browser from the demo corpus shown in the vocabulary count — it ranks candidate words by how often they co-occur with the surviving context. It is a genuine calculation, not a canned list, but it is far simpler than a transformer: treat the shape of the results as the lesson, not the exact percentages.
In one line
MLM turns any raw text into supervised training data by deleting parts of it. Because the blank is surrounded rather than trailing, the model learns deep bidirectional understanding — ideal for classification, retrieval and question answering, and unsuitable for generation. That generative job belongs to causal models, which predict strictly forward.
The variants that improved on BERT
Model
What changed
RoBERTa
More data, longer training, dynamic masking, no next-sentence prediction
ALBERT
Parameter sharing across layers, factorised embeddings
DeBERTa
Disentangled attention for content and position
ELECTRA
Replaced-token detection instead of masking
DistilBERT
Distilled from BERT: 40% smaller, 60% faster, ~97% of the quality
RoBERTa is the instructive one. Same architecture, same objective, and substantially better results — from training longer on more data with larger batches, masking dynamically (a different mask each epoch rather than one fixed at preprocessing), and dropping BERT's next-sentence-prediction objective, which turned out to be unhelpful.
That is a useful general lesson: BERT's original recipe was not optimal, and a large part of the subsequent improvement came from training procedure rather than architecture.
Next-sentence prediction asked whether two segments were adjacent in the original text. It was intended to teach discourse relationships and mostly taught topic matching, which the masked objective already provided.
Masked versus causal, in practice
The split looked permanent for several years — encoders for understanding, decoders for generation. What changed is scale: a large decoder, prompted appropriately, does classification and extraction well enough that the bidirectional advantage stopped being decisive for most applications.
So current practice is roughly:
Need
Reach for
Embeddings for search or clustering
A masked-pretrained encoder, contrastively fine-tuned
Cheap high-volume classification
A fine-tuned small encoder
Token-level labelling
An encoder
Generation of any kind
A decoder
Zero-shot or few-shot on an unknown task
A decoder
Lowest possible inference cost
An encoder
The rows favouring encoders share a shape: a fixed task with labelled data and a cost or latency constraint. That describes a great many production systems, which is why BERT-family models remain very widely deployed despite receiving far less attention.
Practical notes
from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch
tok = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelForMaskedLM.from_pretrained("bert-base-uncased")
text = "The capital of France is [MASK]."
ids = tok(text, return_tensors="pt")
logits = model(**ids).logits
pos = (ids.input_ids[0] == tok.mask_token_id).nonzero(as_tuple=True)[0]
top = logits[0, pos].topk(5).indices[0]
print(tok.convert_ids_to_tokens(top)) # ['paris', 'lyon', 'marseille', ...]
Three things worth knowing when fine-tuning one:
Continued pretraining helps on specialised domains. Running the masked objective on your own unlabelled corpus before supervised fine-tuning often gives a measurable gain, and needs no labels.
The [CLS] representation is not a good sentence embedding out of the box — it was not trained for similarity. Use a model fine-tuned contrastively, or masked mean pooling.
512 tokens is the usual limit. Longer documents need chunking, or a long-context variant such as Longformer.
Why 15 percent, and what the 80/10/10 split is really for
Masked language modelling has two numbers in it that look arbitrary -- a 15 percent masking rate and an 80/10/10 split of what to do with each masked token. Both are answers to specific problems, and both are visible in the arithmetic.
example_01.pyNumPy
import numpy as np
SENT = ["the", "cat", "sat", "on", "the", "soft", "mat", "by", "the", "fire",
"and", "slept", "for", "an", "hour", "or", "so", "quietly", "all",
"afternoon"]
T = len(SENT)
print("A SENTENCE OF %d TOKENS. masked language modelling hides some and" % T)
print("asks the model to restore them, using BOTH sides as context:")
print(" %s" % " ".join(SENT))
print()
print("THE MASKING RATE IS A TRADE-OFF between two things that pull in")
print("opposite directions:")
print("%-10s %14s %20s %22s"
% ("rate", "tokens masked", "training signal", "context left intact"))
for r in (0.05, 0.15, 0.30, 0.50, 0.80):
m = int(round(T * r))
print("%-10s %14d %20s %20.0f%%"
% ("%.0f%%" % (100 * r), m,
"%d prediction%s" % (m, "" if m == 1 else "s"), 100 * (1 - r)))
print(" more masking means more predictions per sentence, so more")
print(" learning signal per forward pass. it also means less context")
print(" left to predict FROM.")
print(" at 80 percent the model is being asked to reconstruct a sentence")
print(" from almost nothing, which is not a language task, it is a")
print(" guessing game. 15 percent is where the two curves cross in")
print(" practice, and it is empirical rather than derived.")
print()
print("AND THE EFFICIENCY COST NOBODY MENTIONS. compare with causal")
print("language modelling, which gets a prediction at EVERY position:")
print("%-34s %16s %20s"
% ("objective", "predictions", "per 1000 tokens"))
print("%-34s %16s %20d"
% ("causal LM", "%d of %d" % (T - 1, T), int(1000 * (T - 1) / T)))
for r in (0.15, 0.30):
print("%-34s %16s %20d"
% ("masked LM at %.0f%%" % (100 * r),
"%d of %d" % (int(round(T * r)), T), int(1000 * r)))
print(" a causal model learns from %d of every %d tokens; a masked model"
% (T - 1, T))
print(" at 15 percent learns from %d. that is roughly %.1fx more"
% (int(round(T * 0.15)), (T - 1) / float(T) / 0.15))
print(" training signal per token of data, and it is the main reason the")
print(" large models everyone uses are causal rather than masked -- not")
print(" because bidirectionality is bad, but because it is expensive.")
print()
rng = np.random.default_rng(3)
print("THE 80/10/10 SPLIT. of the tokens chosen for masking:")
print("%-30s %10s %s" % ("what happens to it", "share", "why"))
print("%-30s %10s %s" % ("replaced with [MASK]", "80%", "the actual task"))
print("%-30s %10s %s" % ("replaced with a RANDOM token", "10%", "see below"))
print("%-30s %10s %s" % ("left unchanged", "10%", "see below"))
print()
print(" the last two are the interesting ones, and both exist to fix a")
print(" MISMATCH between training and use:")
print(" during training, the model sees [MASK] tokens.")
print(" during use, it never does -- [MASK] appears in no real text.")
print(" if 100 percent of masked positions were [MASK], the model could")
print(" learn a shortcut: 'only bother building a real representation")
print(" where I see [MASK], and copy the input everywhere else'. that")
print(" model would score well on the training objective and be useless")
print(" as a feature extractor, which is the only thing it is for.")
print()
print(" the 10 percent RANDOM replacement is what forbids the shortcut:")
chosen = [1, 5, 9]
print("%-8s %-12s %-16s %s" % ("position", "original", "model sees", "must predict"))
for i, (pos, action) in enumerate(zip(chosen, ["[MASK]", "random", "unchanged"])):
seen = {"[MASK]": "[MASK]", "random": "banana",
"unchanged": SENT[pos]}[action]
print("%-8d %-12s %-16s %s" % (pos, SENT[pos], seen, SENT[pos]))
print(" the model cannot tell, from the input alone, whether the token")
print(" in front of it is correct. so it has to build a real")
print(" representation of EVERY position, on the chance that this one is")
print(" the corrupted one.")
print(" the 10 percent unchanged does the same job from the other side:")
print(" sometimes the answer is 'this token is already right', so")
print(" 'predict something different' is not a safe default either.")
print()
print("WHAT THE BIDIRECTIONALITY BUYS. a token's representation can use")
print("both sides, which a causal model cannot do at all:")
print(" the ___ sat on the mat <- left context only")
print(" the ___ sat on the mat, purring and licking its paws")
print(" a causal model predicting position 1 sees only 'the'. a masked")
print(" model sees the entire rest of the sentence, including 'purring',")
print(" which is what actually identifies the animal.")
print(" that is why masked models are better at classification and")
print(" retrieval encoding, and cannot generate text at all: generation")
print(" needs the guarantee that position i never saw i+1, and masked")
print(" language modelling is built on breaking exactly that guarantee.")
print()
print("SO THE TWO OBJECTIVES ARE FOR DIFFERENT JOBS:")
print("%-26s %-22s %s" % ("", "causal LM", "masked LM"))
for row in (("sees", "the left only", "both sides"),
("predictions per token", "1.0", "0.15"),
("can generate", "yes", "no"),
("good for", "generation, chat", "embeddings, classification"),
("typical example", "GPT, Llama", "BERT, and every")):
print("%-26s %-22s %s" % row)
print(" and the last row explains something that looks odd from a")
print(" distance: almost every EMBEDDING model in a RAG pipeline is a")
print(" masked model, while the thing that writes the answer is a causal")
print(" one. a retrieval system usually runs both, for the reasons in")
print(" the two middle rows.")
Output
Questions people ask
Why can BERT not generate text? It has no notion of "next" — its objective assumes surrounding context exists. Iterative mask-filling works poorly.
Is masked language modelling obsolete? Not for embeddings and cheap classification, where encoders remain the practical choice. For general-purpose models, causal pretraining dominates.
Why the 80/10/10 split? Because [MASK] does not exist at fine-tuning time; the random and unchanged cases prevent the model relying on the token's presence.
Should I continue pretraining on my domain? If you have a substantial unlabelled domain corpus, usually yes — it is cheap and often helps.
What is [CLS] for? A position whose final representation is used as a sequence summary for classification.
Can I use a decoder for embeddings? Yes, by pooling hidden states, and purpose-trained embedding models generally do better because they are trained with a similarity objective.
Recap in one screen
Hide about 15% of tokens and predict them from both directions; the representations are the real product.
The 80/10/10 masking split exists because [MASK] never appears at fine-tuning time.
Bidirectional context is why encoders excel at understanding and cannot generate.
Causal modelling trains on every position per pass, which is a large efficiency advantage at scale.
Encoders remain the right tool for embeddings, token labelling and cheap high-volume classification.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
MLM turns any raw text into supervised training data by deleting parts of it. Because the blank is surrounded rather than trailing, the model learns deep bidirectional understanding — ideal for classification, retrieval and question answering, and unsuitable for generation. That generative job belongs to causal models, which predict strictly forward.
What does this module say about “Why It Must Be Bidirectional”?
A causal model reads left to right and predicts what comes next, so it can never look ahead. MLM has no such restriction: the blank sits in the middle , and the model is free to use every token on both sides at once.
What does this module say about “The Curious 80 / 10 / 10 Split”?
BERT masks about 15% of tokens, but does not always insert a literal [MASK] . Of the chosen positions:
Cheat sheet
What is Masked Language Modeling?
Hide words in a sentence and let a bidirectional model guess them back. Click any token to mask it, then watch the model score candidates using context from both sides.
GEN AI · vizlearn.in/gen_ai/masked_language_modeling.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.