Encoder-only vs Decoder-only
BERT and GPT are the same transformer block with one difference: whether a token is allowed to look to its right. Everything else — the training objective, what the model is good for, whether it can write at all — follows from that one triangle of zeros.
Overview
Quick Context
Both families are stacks of the same block: multi-head attention and a feed-forward layer, repeated. The difference is a mask applied inside self-attention — a matrix of minus infinities that deletes some of the attention scores before the softmax.
Delete nothing and every token sees the whole sentence: that is an encoder, and BERT is the famous one. Delete everything above the diagonal and each token sees only itself and what came before: that is a decoder, and GPT is the famous one.
The Architecture
which token is doing the looking
BERT's training objective, in one checkbox
Can It Write?
What Position 5 Can See
—Blue is the token doing the looking. Green is what it is allowed to attend to.
The Attention Mask
The Objective That Allows
Counting It
What Each Is For
Encoder-only — classification, retrieval embeddings, named entities, reranking. Anything where you have the whole text and want to understand it.
Decoder-only — chat, completion, summarising, code. Anything where the output is text you do not have yet.
Encoder-decoder — translation and other strict input-to-output mappings: read bidirectionally, write causally.
BERT vs GPT: A Practical Guide
One triangle of zeros, and everything that follows from it.
Why the mask decides the objective
A model that can see the future cannot be trained to predict it. If position 4 can attend to position 5, then "predict the token at position 5" is answered by copying it, and nothing is learned. So the two masks admit different training tasks.
- Bidirectional → masked language modelling. Hide 15% of the tokens and predict them from both sides. Nothing is being copied because the answer has been removed from the input entirely.
- Causal → next-token prediction. Every position predicts the token after it, and the mask guarantees it cannot cheat. One pass over a sentence supplies a training signal at every position at once, which is what makes it so efficient at scale.
The consequence is the interesting part. Masked language modelling produces excellent representations and no way to generate: there is no "next" for a model where every position already saw everything. Next-token prediction produces a generator whose representations are weaker per parameter, because each token was built without the right-hand context.
Two ways to train on the same architecture
BERT and GPT are both transformers. What separates them is which direction they can look and what they are trained to predict — and everything else follows from that one choice.
| BERT | GPT | |
|---|---|---|
| Blocks | Encoder | Decoder |
| Attention | Bidirectional | Causal — past only |
| Training task | Predict masked tokens | Predict the next token |
| Sees the future | Yes | No |
| Natural at | Understanding | Generating |
| Output used | Token representations | Next-token distribution |
BERT masks about 15% of the tokens and asks the model to fill them in, with the whole rest of the sentence visible in both directions. "The cat sat on the [MASK]" is answered using "cat" and "sat" from the left and any following words from the right.
GPT predicts each token from those before it, with future positions masked out. That constraint is what makes generation possible: at inference the model has only the past, exactly as in training.
Why one generates and the other does not
BERT cannot generate because it has no notion of "next". Its training task assumes the surrounding context exists, so there is nothing to iterate. You can coax text out of it by repeatedly filling masks, and the result is poor.
GPT cannot use future context because it was never allowed to see any. For a classification task where the whole input is available, that is a genuine handicap — the representation of an early token knows nothing about the end of the sentence.
That asymmetry made the split look permanent for several years: encoders for understanding, decoders for generation.
What changed is scale. A sufficiently large decoder, prompted appropriately, does classification, extraction and question answering well enough that the encoder's bidirectional advantage stopped mattering for most applications — and it does them without task-specific fine-tuning. That is why most current large models are decoder-only.
Where each is still the right choice
Use a BERT-style encoder when:
- You need embeddings for search, clustering or similarity. Sentence-BERT and its successors dominate here.
- You are doing token classification — named entities, part-of-speech tagging — where bidirectional context helps and the whole input is available.
- You want a small, fast, cheap classifier. A fine-tuned DistilBERT runs on a CPU in milliseconds; a large decoder does not.
- You have a fixed task with labelled data. Fine-tuning a 110M-parameter encoder often matches or beats prompting a much larger model, at a fraction of the cost.
Use a GPT-style decoder when:
- The output is text — writing, summarising, translating, answering.
- The task is open-ended or not known in advance.
- You have few or no labels and want to work by prompting.
- You need several tasks from one model without training a head per task.
One matrix of zeros and infinities
BERT and GPT share almost all their architecture. The difference is a mask applied to the attention scores, and this builds both so you can see what each one can and cannot look at.
Experiments to try
- Look around from the middle. With bidirectional attention selected and position 5, the whole sentence lights up: 8 tokens visible, 2 of them to the right.
- Switch to causal. The same position now sees 6 tokens and none to the right, and the mask grid loses its whole upper triangle — 36 visible pairs out of 64 instead of all 64.
- Walk the query along. Drag Look From Position from 0 to 7 with causal attention on. Position 0 sees exactly one token — itself. That is why the first token of a prompt is the hardest thing a decoder ever has to represent.
- Try the masked objective. Switch back to bidirectional, put the query on position 5 and tick Mask That Token Out. The word is replaced by [MASK] and the model has "on the ___ mat" to work with — both sides. That is BERT's whole training task.
- Now mask under causal attention. Same checkbox, causal mode: the model has "The cat sat on the ___" and no idea that "mat" follows. Same sentence, strictly less information, which is exactly the trade.
- Try to generate. Press Generate Next in causal mode and the sentence extends one token at a time, each new token attending only to what precedes it. Switch to bidirectional and the button refuses: an encoder has no notion of next.
Which one to reach for
- Classifying, tagging, reranking, embedding for search? Encoder. You already have the whole text; a model that reads it bidirectionally will represent it better and is far cheaper to run — a BERT-sized encoder is hundreds of times smaller than a frontier decoder.
- Producing text? Decoder. Nothing else can.
- Strict input-to-output rewriting, like translation? Encoder-decoder, which reads with full attention and writes causally, plus cross-attention between the two. T5 and the original transformer are this shape.
The industry has drifted decisively towards decoder-only for general systems, because one model that can be prompted to do anything beats a family of specialists — but for a high-volume classifier or a retrieval encoder, the small bidirectional model is still usually the right engineering answer.
The short of it
Encoder-only and decoder-only transformers differ by one thing: whether the attention mask deletes the upper triangle. Bidirectional attention lets every token see the whole sequence, which rules out next-token prediction and demands a masked objective instead, and yields strong representations with no ability to generate. Causal attention lets each token see only what precedes it, which makes next-token prediction both possible and enormously efficient, and yields a generator. Pick by the task rather than by fashion: text out means a decoder, and understanding text you already have is still cheaper and often better with an encoder.
The third family: encoder-decoder
T5, BART and the original transformer use both stacks: an encoder that reads the input bidirectionally, and a decoder that generates while attending to the encoder's output through cross-attention.
That structure suits tasks with a clear input and a distinct output — translation, summarisation, grammatical correction. T5's framing was to treat every task as text-to-text, with a prefix saying which task it is.
| Family | Example models |
|---|---|
| Encoder-only | BERT, RoBERTa, DeBERTa, DistilBERT, E5 |
| Decoder-only | GPT, Llama, Mistral, Claude, Gemini |
| Encoder-decoder | T5, BART, mT5, Flan-T5 |
Encoder-decoder models are less prominent now, largely because decoder-only models scaled better and can do the same tasks by prompting. They remain strong when the task is genuinely a fixed transformation and you can fine-tune.
The training details that mattered
BERT's masked language modelling replaces 15% of tokens: 80% with [MASK], 10% with a random token, 10% left unchanged. The randomisation exists because [MASK] never appears at fine-tuning time, and a model trained only on it would be mismatched.
Next sentence prediction was BERT's second objective — predicting whether two sentences were adjacent. RoBERTa showed it was unhelpful and removed it, along with training longer on more data, which improved results substantially. It is a useful reminder that the original recipe was not optimal.
GPT's next-token prediction needs no labelling at all: any text is training data. That simplicity is a large part of why decoder-only models scaled — the training set is the internet rather than an annotated corpus.
Instruction tuning and RLHF are what turn a raw next-token predictor into something that follows instructions. A base GPT-style model completes text; it does not answer questions until it has been fine-tuned to.
Questions people ask
Which is better? Neither — they are shaped for different jobs. For embeddings and cheap fixed-task classification, an encoder. For anything generative or open-ended, a decoder.
Can I use GPT-style models for embeddings? Yes, by pooling hidden states, and purpose-trained embedding models generally do better because they are trained with a similarity objective.
Why does BERT have [CLS]? It is a position whose final representation is used as a summary of the whole sequence for classification.
Is fine-tuning BERT still worth doing? Very much so, when you have labelled data and a fixed task. It is far cheaper to run than prompting a large model, and often as accurate.
What makes RoBERTa different? Same architecture, better training: more data, longer, larger batches, dynamic masking, and no next-sentence prediction.
Do decoders really understand as well as encoders? At sufficient scale, they perform comparably on most understanding benchmarks despite the one-directional constraint. Bidirectional attention is an advantage that scale largely compensates for.
Recap in one screen
- Same architecture; the difference is bidirectional masked prediction versus causal next-token prediction.
- BERT sees both directions and cannot generate; GPT sees only the past and therefore can.
- Encoders are the right tool for embeddings and cheap fixed-task classification.
- Decoders are the right tool for generation and for open-ended tasks with few labels.
- Encoder-decoder models suit fixed input-to-output transformations, and lost prominence as decoders scaled.