Encoder-only vs Decoder-only

Quick Context

Both families are stacks of the same block: multi-head attention and a feed-forward layer, repeated. The gap is a mask applied inside self-attention. A matrix of minus infinities that deletes some of the attention scores before the softmax.

Delete nothing and each token sees the whole sentence. That is an encoder, and BERT is the famous one. Delete all of it above the diagonal and each token sees only itself and what came before. That is a decoder, and GPT is the famous one. The mask decides the training objective. A bidirectional model cannot be trained to predict the next token (it would see the answer). So BERT uses masked language modelling. Hide 15% of the tokens and predict them from both sides. A causal model trains on next-token prediction, which supplies a training sign at each spot in one pass and scales to trillions of tokens. The result: BERT gives context embeddings that see the whole sentence and excels at classification, NER, and extractive QA (fine-tune a head on top). GPT makes text token by token and excels at open-ended generation, summarisation. And in-context learning (prompt it, no fine-tuning needed). This page shows the attention mask for both designs, shows how the same sentence gives different forms, and covers the encoder-decoder models (T5, BART) that combine both halves.

Why the mask decides the objective

A model that can see the future cannot be trained to predict it. If spot 4 can attend to spot 5. Then "predict the token at spot 5" is answered by copying it, and nothing is learned. So the two masks admit different training tasks.

• Bidirectional → masked language modelling. Hide 15% of the tokens and predict them from both sides. Nothing is being copied because the answer has been removed from the input fully.
• Causal → next-token prediction. Each spot predicts the token after it. And the mask guarantees it cannot cheat. One pass over a sentence supplies a training sign at each spot at once. This is what makes it so fast at scale.

The result is the notable part. Masked language modelling gives excellent forms and no way to generate. There is no "next" for a model where each spot now saw all of it. Next-token prediction gives a generator whose forms are weaker per parameter, because each token was built without the right-hand context.

Two ways to train on the same design

BERT and GPT are both transformers. What separates them is which way they can look and what they are trained to predict. And all of it else follows from that one choice.

BERT masks about 15% of the tokens and asks the model to fill them in, with the whole rest of the sentence visible in both ways. "The cat sat on the [MASK]" is answered using "cat" and "sat" from the left and any below words from the right.

GPT predicts each token from those before it, with future spots masked out. That constraint is what makes generation doable: at inference the model has only the past, just as in training.

Why one makes and the other does not

BERT cannot generate because it has no notion of "next". Its training task assumes the nearby context exists, so there is nothing to iterate. You can coax text out of it by repeatedly filling masks. And the result is poor.

GPT cannot use future context because it was never allowed to see any. For a classification task where the whole input is available, that is a genuine handicap. The form of an early token knows nothing about the end of the sentence.

That asymmetry made the split look permanent for a few years: encoders for grasp, decoders for generation.

What changed is scale. A sufficiently large decoder, prompted rightly, does classification, extraction and question answering well enough that the encoder's bidirectional edge stopped mattering for most uses. And it does them without task-set fine-tuning. That is why most current large models are decoder-only.

Where each is still the right choice

Use a BERT-style encoder when:

• You need embeddings for search, clustering or likeness. Sentence-BERT and its successors dominate here.
• You are doing token classification. Named entities, part-of-speech tagging. Where bidirectional context helps and the whole input is available.
• You want a small, fast, cheap classifier. A fine-tuned DistilBERT runs on a CPU in milliseconds; a large decoder does not.
• You have a fixed task with labelled data. Fine-tuning a 110M-parameter encoder often matches or beats prompting a much larger model, at a fraction of the cost.

Use a GPT-style decoder when:

• The output is text — writing, summarising, translating, answering.
• The task is open-ended or not known in advance.
• You have few or no labels and want to work by prompting.
• You need a few tasks from one model without training a head per task.

One matrix of zeros and infinities

BERT and GPT share almost all their design. The gap is a mask applied to the attention scores. And this builds both so you can see what each one can and cannot look at.

import numpy as np

rng = np.random.default_rng(0)
words = ["the", "bank", "raised", "its", "rates"]
T, D = len(words), 8
E = rng.normal(0, 1.0, (T, D))
Wq, Wk, Wv = (rng.normal(0, 0.5, (D, D)) for _ in range(3))

def softmax(z):
    e = np.exp(z - z.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

scores = (E @ Wq) @ (E @ Wk).T / np.sqrt(D)

print("BIDIRECTIONAL (BERT). no mask -- every token sees every token:")
A_bi = softmax(scores)
print("%8s %s" % ("", "".join("%9s" % w for w in words)))
for i, w in enumerate(words):
    print("%8s %s" % (w, "".join("%9.4f" % a for a in A_bi[i])))
print()

Building on the same setup, the next block shows the left cases.

import numpy as np

causal = np.triu(np.ones((T, T)), k=1) * -np.inf
causal = np.nan_to_num(causal, neginf=-np.inf)
print("CAUSAL (GPT). add this mask to the scores before the softmax:")
print("%8s %s" % ("", "".join("%9s" % w for w in words)))
for i, w in enumerate(words):
    print("%8s %s" % (w, "".join("%9s" % ("-inf" if np.isinf(causal[i][j]) else "0")
                                 for j in range(T))))
A_causal = softmax(scores + causal)
print()
print("exp(-inf) = 0, so those positions get exactly zero weight:")
print("%8s %s" % ("", "".join("%9s" % w for w in words)))
for i, w in enumerate(words):
    print("%8s %s" % (w, "".join("%9.4f" % a for a in A_causal[i])))
print("   row sums:", np.round(A_causal.sum(1), 6))
print()
print("   the first row has one option and must give it all 1.0. the last")
print("   row can see everything. each row renormalises over what is left.")
print()

print("WHAT THAT CHANGES. take 'bank', which is ambiguous until you reach")
print("'rates':")
i = words.index("bank")
print("   bidirectional -- it can see 'rates' with weight %.4f"
      % A_bi[i][words.index("rates")])
print("   causal        -- it can see 'rates' with weight %.4f"
      % A_causal[i][words.index("rates")])
print("   in the causal model the representation of 'bank' at position 1 is")
print("   built without ever seeing the rest of the sentence. it cannot be")
print("   disambiguated, because the disambiguating word has not arrived.")
print()

The below block covers the left steps.

import numpy as np

print("WHY GPT ACCEPTS THAT COST. the mask is what makes next-token")
print("prediction a valid training objective:")
print("   without a mask, predicting token t is trivial -- token t is in")
print("   the input. the model would learn to copy and nothing else.")
print("   with the mask, position t only sees tokens 0..t, so predicting")
print("   t+1 is a real problem. and EVERY position is a training example")
print("   at once:")
print("      a %d-token sequence gives %d predictions in one forward pass."
      % (T, T - 1))
print("   that is why generative pretraining scales: no labels, and the")
print("   whole sequence is supervision.")
print()

print("HOW BERT TRAINS INSTEAD. it cannot predict the next token -- it can")
print("see it. so it corrupts the input and predicts the corruption:")
masked = list(words)
masked[2] = "[MASK]"
print("   input : %s" % " ".join(masked))
print("   target: predict 'raised' at position 2, using BOTH sides")
print("   about 15 percent of tokens are masked, so a %d-token sequence"
      % T)
print("   yields roughly %.1f training signals rather than %d." % (0.15 * T, T - 1))
print("   BERT gets a better representation per token and far less")

The next block goes on with the left demonstrations.

print("   supervision per sequence, which is a real part of why generative")
print("   models scaled further.")
print()

print("THE SUMMARY, as a table:")
rows = [("attention mask", "none", "causal (lower triangular)"),
        ("a token sees", "all positions", "itself and earlier"),
        ("pretraining", "masked-token prediction", "next-token prediction"),
        ("signals per sequence", "~15% of tokens", "every token"),
        ("natural at", "classification, tagging,", "generation, completion,"),
        ("", "retrieval, span extraction", "anything autoregressive"),
        ("generation", "awkward -- no natural order", "native")]
print("%22s %28s %28s" % ("", "BERT (encoder)", "GPT (decoder)"))
for a, b, c in rows:
    print("%22s %28s %28s" % (a, b, c))
print()
print("everything else -- the attention arithmetic, the feed-forward blocks,")
print("the residuals, the layer norms -- is the same in both. the")
print("architectural difference really is that one matrix.")
print()
print("and the practical consequence: a bidirectional encoder is usually the")
print("better choice when you have the whole input and want to LABEL it. a")
print("causal decoder is the only choice when you need to PRODUCE text. the")
print("fact that decoders now also do the labelling well is a result about")
print("scale, not about the mask.")

Tests to try

1. Look around from the middle. With bidirectional attention selected and spot 5, the whole sentence lights up: 8 tokens visible, 2 of them to the right.
2. Switch to causal. The same spot now sees 6 tokens and none to the right. And the mask grid loses its whole upper triangle — 36 visible pairs out of 64 instead of all 64.
3. Walk the query along. Drag Look From Spot from 0 to 7 with causal attention on. Spot 0 sees just one token — itself. That is why the first token of a prompt is the hardest thing a decoder ever has to represent.
4. Try the masked objective. Switch back to bidirectional, put the query on spot 5 and tick Mask That Token Out. The word is replaced by [MASK] and the model has "on the ___ mat" to work with — both sides. That is BERT's whole training task.
5. Now mask under causal attention. Same checkbox, causal mode. The model has "The cat sat on the ___" and no idea that "mat" follows. Same sentence, strictly less data, which is just the trade.
6. Try to generate. Press Generate Next in causal mode and the sentence extends one token at a time, each new token attending only to what precedes it. Switch to bidirectional and the button refuses: an encoder has no notion of next.

Which one to reach for

• Classifying, tagging, reranking, embedding for search? Encoder. You now have the whole text; a model that reads it bidirectionally will represent it better and is far cheaper to run. A BERT-sized encoder is hundreds of times smaller than a frontier decoder.
• Giving text? Decoder. Nothing else can.
• Strict input-to-output rewriting, like translation? Encoder-decoder, which reads with full attention and writes causally, plus cross-attention between the two. T5 and the first transformer are this shape.

The industry has drifted decisively towards decoder-only for general systems, because one model that can be prompted to do a thing beats a family of specialists. But for a high-volume classifier or a retrieval encoder, the small bidirectional model is still often the right engineering answer.

The short of it

Encoder-only and decoder-only transformers differ by one thing: whether the attention mask deletes the upper triangle. Bidirectional attention lets each token see the whole sequence, which rules out next-token prediction and demands a masked objective instead, and yields strong forms with no ability to generate. Causal attention lets each token see only what precedes it. This makes next-token prediction both doable and enormously fast, and yields a generator. Pick by the task rather than by fashion. Text out means a decoder, and grasp text you now have is still cheaper and often better with an encoder.

The third family: encoder-decoder

T5, BART and the first transformer use both stacks: an encoder that reads the input bidirectionally. And a decoder that makes while attending to the encoder's output through cross-attention.

That structure suits tasks with a clear input and a distinct output. Translation, summarisation, grammatical correction. T5's framing was to treat each task as text-to-text, with a prefix saying which task it is.

Encoder-decoder models are less prominent now, largely because decoder-only models scaled better and can do the same tasks by prompting. They remain strong when the task is truly a fixed transformation and you can fine-tune.

The training details that mattered

BERT's masked language modelling replaces 15% of tokens: 80% with [MASK], 10% with a random token, 10% left unchanged. The randomisation exists because [MASK] never appears at fine-tuning time. And a model trained only on it would be mismatched.

Next sentence prediction was BERT's second objective — predicting whether two sentences were adjacent. RoBERTa showed it was unhelpful and removed it, along with training longer on more data, which improved results much. It is a useful reminder that the first recipe was not best.

GPT's next-token prediction needs no labelling at all: any text is training data. That ease is a large part of why decoder-only models scaled. The training set is the internet rather than an annotated corpus.

Cue tuning and RLHF are what turn a raw next-token predictor into a thing that follows cues. A base GPT-style model completes text; it does not answer questions until it has been fine-tuned to.

Questions people ask

Which is better? Neither — they are shaped for different jobs. For embeddings and cheap fixed-task classification, an encoder. For a thing generative or open-ended, a decoder.

Can I use GPT-style models for embeddings? Yes, by pooling hidden states, and purpose-trained embedding models often do better. This is because they are trained with a likeness objective.

Why does BERT have [CLS]? It is a spot whose final form is used as a summary of the whole sequence for classification.

Is fine-tuning BERT still worth doing? Very much so. When you have labelled data and a fixed task. It is far cheaper to run than prompting a large model, and often as exact.

What makes RoBERTa different? Same design, better training: more data, longer, larger batches, dynamic masking. And no next-sentence prediction.

Do decoders really understand and encoders? At enough scale, they perform comparably on most grasp benchmarks despite the one-one-way constraint. Bidirectional attention is an edge that scale largely compensates for.

Recap in one screen

• Same design; the gap is bidirectional masked prediction versus causal next-token prediction.
• BERT sees both ways and cannot generate; GPT sees only the past and so can.
• Encoders are the right tool for embeddings and cheap fixed-task classification.
• Decoders are the right tool for generation and for open-ended tasks with few labels.
• Encoder-decoder models suit fixed input-to-output transformations, and lost prominence as decoders scaled.