What is a Bidirectional Layer?

How the two passes combine

A bidirectional layer runs two entirely separate recurrent cells with their own weights. The forward cell reads left to right; the backward cell reads the same sequence right to left. At each position their hidden states are concatenated:

h_t = [h→_t ; h←_t]

Two consequences follow immediately. The output at each position is twice as wide, so the next layer’s input size doubles — a frequent shape-mismatch bug. And there are twice the parameters, since nothing is shared between the two directions.

Note the passes are independent: the backward cell does not see the forward cell’s states. They are computed separately and only joined at the end, which also means they can run in parallel.

Reading the sequence twice

A forward LSTM at word 5 knows words 1 to 5. A bidirectional layer runs a second, independent LSTM backwards — from the end to the beginning — and concatenates the two hidden states at each position.

hᵗ = [hᵗ^→ ; hᵗ^←]

So the representation at word 5 now contains what came before it and what came after. The output width doubles: 256 hidden units in each direction gives 512-dimensional outputs.

Why it helps is easiest to see with an ambiguous word:

"The bank was steep and muddy."

At "bank", a forward-only model has seen "the" and nothing else — it cannot yet know which sense is meant. The backward pass has seen "steep and muddy" and resolves it immediately. Neither direction alone is sufficient; together they are.

The same argument applies to named entities ("Washington" as a person or a place depends on what follows), part-of-speech tagging, and coreference.

The hard constraint

Bidirectional layers require the entire sequence to be available before processing starts. That rules them out for two important cases:

Generation. Predicting the next word cannot use the next word. A bidirectional language model would have access to the answer, which makes training trivial and inference impossible.

Streaming. Live transcription, real-time translation and any incremental system cannot wait for the end of the input.

So the rule is clear: bidirectional for understanding a complete input, unidirectional for generating or streaming.

That last pair is worth noting: an encoder-decoder model is frequently bidirectional in the encoder and causal in the decoder, because the source sentence is complete and the target is being produced.

Using one

lstm = nn.LSTM(300, 256, batch_first=True, bidirectional=True)
out, (h, c) = lstm(x)

out.shape          # (batch, seq, 512)  -- 256 forward + 256 backward
h.shape            # (2, batch, 256)    -- one final state per direction

Two details cause most of the confusion.

The output is twice as wide. Any layer after it must expect 2 × hidden_size. Forgetting this is the most common bidirectional bug, and it produces an immediate shape error.

The final states are not simply the last row of out. The forward direction's final state corresponds to the last token; the backward direction's corresponds to the first. For classification, concatenate the two final states, or pool over out — do not take out[:, -1, :], which mixes the forward state at the end with the backward state at the end (which has seen only the last token).

h_fwd, h_bwd = h[0], h[1]
summary = torch.cat([h_fwd, h_bwd], dim=-1)     # correct for classification

Reading the sequence twice, in both directions

A bidirectional layer runs two independent RNNs and concatenates them. That doubles the parameters, gives every position full context, and makes the layer unusable for generation -- all for the same reason.

import numpy as np

rng = np.random.default_rng(0)
words = ["the", "bank", "by", "the", "river", "flooded"]
T, X, H = len(words), 4, 5
seq = rng.normal(0, 1.0, (T, X))

Wxf = rng.normal(0, 0.5, (X, H)); Whf = rng.normal(0, 0.5, (H, H))
Wxb = rng.normal(0, 0.5, (X, H)); Whb = rng.normal(0, 0.5, (H, H))

def run(seq, Wx, Wh):
    h = np.zeros(H)
    out = []
    for x in seq:
        h = np.tanh(x @ Wx + h @ Wh)
        out.append(h.copy())
    return np.array(out)

fwd = run(seq, Wxf, Whf)                    # left to right
bwd = run(seq[::-1], Wxb, Whb)[::-1]        # right to left, then re-align
both = np.concatenate([fwd, bwd], axis=1)

print("two SEPARATE RNNs with their own weights, run over the same tokens:")
print("   forward  reads %s" % " -> ".join(words))
print("   backward reads %s" % " -> ".join(words[::-1]))
print()
print("%10s %14s %14s %14s"
      % ("position", "forward h[0]", "backward h[0]", "concatenated"))
for t, w in enumerate(words):
    print("%10s %14.4f %14.4f %14s"
          % (w, fwd[t][0], bwd[t][0], "%d dims" % both.shape[1]))
print()
print("the re-alignment matters and is the usual bug: the backward pass is")
print("computed on the reversed sequence and must be flipped back, so that")
print("position t's forward and backward states describe the SAME token.")
print()

print("WHAT EACH DIRECTION KNOWS at position 1 ('bank'):")
print("   forward  has seen: %s" % " ".join(words[:2]))
print("   backward has seen: %s" % " ".join(words[1:][::-1]))
print("   together: the whole sentence.")
print()
print("measure it -- change the LAST word and see which states move:")
alt = seq.copy()
alt[-1] = rng.normal(0, 1.0, X)
f2 = run(alt, Wxf, Whf)
b2 = run(alt[::-1], Wxb, Whb)[::-1]
print("%10s %18s %18s" % ("position", "forward changed?", "backward changed?"))
for t, w in enumerate(words):
    df = np.linalg.norm(fwd[t] - f2[t])
    db = np.linalg.norm(bwd[t] - b2[t])
    print("%10s %18.5f %18.5f" % (w, df, db))
print("   the forward states before the last position are untouched -- they")
print("   were computed before that word arrived. the backward states all")
print("   moved, because for them it was the FIRST word.")
print("   that column is the whole value of the layer.")
print()

print("THE COST. parameters and compute both double:")
uni = X * H + H * H + H
print("   unidirectional : %d parameters, output width %d" % (uni, H))
print("   bidirectional  : %d parameters, output width %d" % (2 * uni, 2 * H))
print("   and the next layer's input is twice as wide, so ITS parameters")
print("   grow too. a stack of bidirectional layers costs more than 2x.")
print()

print("THE RESTRICTION, which is not a detail. the backward pass needs the")
print("whole sequence before it can produce position 0's state:")
print("   generation      -- impossible. you would need the tokens you have")
print("                      not generated yet.")
print("   streaming       -- impossible. no output until the input ends.")
print("   classification  -- fine. you have the whole input already.")
print("   tagging (NER,   -- fine, and this is where it shines: deciding")
print("   POS)               whether 'bank' is a noun or a verb needs the")
print("                      words after it as much as the ones before.")
print()
print("that trade is exactly the BERT/GPT distinction one level down. a")
print("bidirectional encoder and a causal decoder are answering the same")
print("question -- may this position see the future? -- and the answer")
print("decides what the model can be used for.")
print()

print("the practical shape, for a tagging model:")
print("   embed -> BiLSTM -> concat(forward, backward) -> per-token classifier")
print("   output width %d -> %d classes" % (2 * H, 9))
print("   note the classifier sees both halves for every token. that is the")
print("   arrangement, and it was the standard for named-entity recognition")
print("   for most of a decade before encoders replaced it.")

Guided experiments

1. Watch both passes. Press Simulate Dual Passes and follow the two chains. One advances left to right, the other right to left, and they meet only where the outputs concatenate.
2. Confirm the doubling. Press Parameters and compare with a unidirectional cell of the same Hidden Size (Per Dir). Exactly twice as many — two independent sets of weights.
3. Check the output width. Set Hidden Size (Per Dir) to 20 and note the layer emits 40 values per position. The next layer must be built for 40, not 20.
4. Grow the input. Raise Input Vector and watch both directions scale together. The two passes are symmetric in cost; nothing is shared or saved.

When you cannot use it

A bidirectional layer requires the entire sequence before it can produce any output, because the backward pass starts at the end. That rules it out whenever the future genuinely is not available:

• Language modelling and text generation. Predicting the next token while having already read it is not a prediction. A bidirectional model here achieves perfect accuracy and learns nothing — the answer leaks in through the backward pass.
• Real-time and streaming applications. Live transcription cannot wait for the end of the utterance.
• Autoregressive decoding of any kind, where output is produced one token at a time.

Where the full sequence is available — classification, tagging, named entity recognition, and the encoder half of a translation model — bidirectionality is close to free accuracy. This is exactly the split between BERT, which is bidirectional and cannot generate, and GPT, which is unidirectional and can.

Traps worth knowing

• Forgetting the output doubles. The most common bug: the following layer is sized for h instead of 2h and the shapes do not match.
• Using it for generation. Training looks superb and the model is useless, because it has been shown the answer.
• Padding without masking. Worse here than in a unidirectional model — the backward pass starts in the padding, so an unmasked model begins reading noise.
• Assuming it doubles quality. It doubles cost. The accuracy gain is real but usually a few points, and on some tasks a wider unidirectional layer is the better use of the same parameters.

The short version

A bidirectional layer runs independent forward and backward recurrent passes and concatenates them, so every position is represented with context from both sides — at twice the parameters and twice the output width. It is the right default for classification and tagging, where the whole sequence is available, and impossible for generation or streaming, where the backward pass would be reading the future it is meant to predict.

The cost

Twice the computation and twice the parameters. Two independent LSTMs, each with its own four weight matrices. For 300-dimensional input and 256 hidden units, about 1.15 million parameters instead of 574,000.

Twice the memory during training, since both passes' activations must be stored.

No latency benefit from parallelism. The two directions are independent of each other and can run concurrently, but each is still strictly sequential internally.

Full input required, which is the real cost rather than the arithmetic.

Whether the accuracy gain justifies it depends on the task. For tagging and span extraction, where the right context is genuinely necessary, the gain is usually substantial. For document-level topic classification, where the signal is distributed and redundant, it is often marginal.

The transformer equivalent

Bidirectionality in transformers is not a separate layer — it is the default, and unidirectionality is what has to be added.

An encoder's self-attention lets every position attend to every other, in both directions, with no extra machinery. That is exactly what BERT is: a bidirectional encoder, and its masked-language-model training objective exists precisely because bidirectional attention makes next-token prediction trivial.

A decoder adds a causal mask that sets attention scores for future positions to negative infinity, so their weights become zero. Generation becomes possible because the model genuinely cannot see ahead.

So "bidirectional layer" is largely a recurrent-era concept. The idea survives; the implementation is a mask rather than a second network.

Questions people ask

Can I use a bidirectional layer for language modelling? No — it would see the token it is meant to predict.

Does the backward LSTM share weights with the forward one? No, they are entirely separate parameter sets.

Should I concatenate or sum the two directions? Concatenate, which is the standard and keeps both intact. Summing halves the width and loses information.

Is BERT a bidirectional LSTM? No — it is a bidirectional transformer encoder. The word "bidirectional" in its name refers to unmasked attention.

Does bidirectionality help every task? No. It helps most where right context disambiguates — tagging, entity recognition, span extraction — and least on tasks with redundant global signal.

Can I stack bidirectional layers? Yes, and each layer's doubled output feeds the next. Two is usually the practical limit.

Recap in one screen

• A bidirectional layer runs a second recurrent pass backwards and concatenates the states.
• Each position then has both left and right context, which resolves ambiguity that neither direction can alone.
• It requires the complete input, so it is unusable for generation and streaming.
• Output width doubles, and the two final states correspond to opposite ends of the sequence.
• In transformers, bidirectionality is the default and a causal mask is what creates the unidirectional case.