Hide words in a sentence and let a bidirectional model guess them back. Click any token to mask it, then watch the model score candidates using context from both sides.
Input Sequence
CLICK TO MASK
← left contextboth directions feed the predictionright context →
Model Predictions
Click a token above to mask it
Masked Language Modeling: Learning by Filling Blanks
Masked Language Modeling (MLM) is the pretraining objective behind BERT and its descendants. The recipe is deceptively simple: take ordinary text, hide a fraction of the tokens, and train the model to reconstruct exactly what was hidden. No human labels required — the text supplies its own answer key.
Why It Must Be Bidirectional
A causal model reads left to right and predicts what comes next, so it can never look ahead. MLM has no such restriction: the blank sits in the middle, and the model is free to use every token on both sides at once.
Mask soft in "the cat sat on the ___ mat" and the left context narrows it to a modifier while the right context (mat) rules out most adjectives. Neither side alone is enough. That is why MLM produces such strong understanding representations — and why it cannot be used to generate text.
The Curious 80 / 10 / 10 Split
BERT masks about 15% of tokens, but does not always insert a literal [MASK]. Of the chosen positions:
80% become [MASK] — the standard fill-in-the-blank case.
10% are replaced by a random word — forcing the model to notice when a token does not fit, rather than trusting whatever it sees.
10% are left unchanged — but still predicted, so the model keeps a useful representation of every token, not just masked ones.
The reason is a train/test mismatch: [MASK] never appears during fine-tuning or inference. If every masked slot were a literal mask token, the model would learn features that only activate on a symbol it will never see again.
Masking Rate Is a Trade-off
Mask too little and each sentence teaches the model almost nothing — training is slow because most positions carry no loss. Mask too much and you destroy the very context needed to make a prediction. Around 15% is the long-standing sweet spot, though later work has shown much higher rates can work with large enough models. Drag the rate slider and watch the sentence degrade: at 50% the remaining context often cannot pin the answer down.
Interactive Exploration Guide
Mask a single strongly-constrained token such as mat. The top candidate should dominate — the surrounding words leave little freedom.
Now mask an adjective like soft. Confidence spreads across several plausible words: the context permits a whole family of answers.
Load the "bank / river" preset and mask bank. The words after it are what disambiguate the sense — exactly the information a left-to-right model would not yet have.
Mask two adjacent tokens. Accuracy drops sharply, because MLM predicts each blank independently and cannot coordinate between them — a genuine known limitation of the objective.
About the model in this lab
Predictions come from a small context-scoring model built live in your browser from the demo corpus shown in the vocabulary count — it ranks candidate words by how often they co-occur with the surviving context. It is a genuine calculation, not a canned list, but it is far simpler than a transformer: treat the shape of the results as the lesson, not the exact percentages.
Key Takeaway
MLM turns any raw text into supervised training data by deleting parts of it. Because the blank is surrounded rather than trailing, the model learns deep bidirectional understanding — ideal for classification, retrieval and question answering, and unsuitable for generation. That generative job belongs to causal models, which predict strictly forward.
Cheat sheet
What is Masked Language Modeling?
Hide words in a sentence and let a bidirectional model guess them back. Click any token to mask it, then watch the model score candidates using context from both sides.
GEN AI · vizlearn.in/gen_ai/masked_language_modeling.html