What is a Bidirectional Layer?

Why one direction is not enough

A forward RNN at spot t has seen tokens 1 to t and nothing after. That is a real handicap, because disambiguation often depends on what comes next.

Take “The bank was steep and muddy.” At the word bank a forward-only model has seen only “The”, and must commit to a form before steep and muddy arrives to settle the meaning. A backward pass has that data at once.

The fix is to run two passes — one forward, one backward — and concatenate their outputs at each spot. The forward pass gives a hidden state that encodes all of it to the left; the backward pass gives a hidden state that encodes all of it to the right. Stacking the two gives each spot a form that sees the entire sequence.

This is the idea behind the bidirectional layer. It doubles the hidden dimension, because it concatenates two free states. And it doubles the parameter count, because the forward and backward cells learn separate weights. The precision gain is large for tasks like named-entity recognition, part-of-speech tagging, sentiment classification and question answering. Any task where the label at spot t depends on tokens that come after t. The cost is that the full sequence must be available before the layer can run. This rules out autoregressive generation: you cannot predict the next word with a backward pass that needs words you have not made yet. That constraint is why GPT-style models use only a forward pass and BERT-style models use bidirectional layers.

How the two passes combine

A bidirectional layer runs two fully separate recurrent cells with their own weights. The forward cell reads left to right; the backward cell reads the same sequence right to left. At each spot their hidden states are concatenated:

h_t = [h→_t . H←_t]

Two results follow at once. The output at each spot is twice as wide, so the next layer’s input size doubles. A frequent shape-mismatch bug. And there are twice the parameters, since nothing is shared between the two directions.

Note the passes are free: the backward cell does not see the forward cell’s states. They are computed apart and only joined at the end. This also means they can run in parallel.

Reading the sequence twice

A forward LSTM at word 5 knows words 1 to 5. A bidirectional layer runs a second, free LSTM backwards. From the end to the beginning. And concatenates the two hidden states at each spot.

hᵗ = [hᵗ^→ . Hᵗ^←]

So the form at word 5 now holds what came before it and what came after. The output width doubles: 256 hidden units in each direction gives 512-dimensional outputs.

Why it helps is easiest to see with an ambiguous word:

"The bank was steep and muddy."

At "bank", a forward-only model has seen "the" and nothing else. It cannot yet know which sense is meant. The backward pass has seen "steep and muddy" and resolves it at once. Neither direction alone is enough; together they are.

The same argument applies to named entities ("Washington" as a person or a place depends on what follows), part-of-speech tagging, and coreference.

The hard constraint

Bidirectional layers need the entire sequence to be available before handling starts. That rules them out for two key cases:

Generation. Predicting the next word cannot use the next word. A bidirectional language model would have access to the answer. This makes training trivial and inference not doable.

Streaming. Live transcription, real-time translation and any incremental system cannot wait for the end of the input.

So the rule is clear: bidirectional for grasp a complete input, unidirectional for making or streaming.

That last pair is worth noting: an encoder-decoder model is often bidirectional in the encoder and causal in the decoder,. This is because the source sentence is complete and the target is being gave.

Using one

lstm = nn.LSTM(300, 256, batch_first=True, bidirectional=True)
out, (h, c) = lstm(x)

out.shape          # (batch, seq, 512)  -- 256 forward + 256 backward
h.shape            # (2, batch, 256)    -- one final state per direction

Two details cause most of the confusion.

The output is twice as wide. Any layer after it must expect 2 × hidden_size. Forgetting this is the most common bidirectional bug, and it gives an immediate shape error.

The final states are not simply the last row of out. The forward direction's final state corresponds to the last token; the backward direction's corresponds to the first. For classification, concatenate the two final states, or pool over out. Do not take out[:, -1, :]. This mixes the forward state at the end with the backward state at the end (which has seen only the last token).

h_fwd, h_bwd = h[0], h[1]
summary = torch.cat([h_fwd, h_bwd], dim=-1)     # correct for classification

Reading the sequence twice, in both directions

A bidirectional layer runs two free RNNs and concatenates them. That doubles the parameters, gives each spot full context, and makes the layer unusable for generation. All for the same reason.

import numpy as np

rng = np.random.default_rng(0)
words = ["the", "bank", "by", "the", "river", "flooded"]
T, X, H = len(words), 4, 5
seq = rng.normal(0, 1.0, (T, X))

Wxf = rng.normal(0, 0.5, (X, H)); Whf = rng.normal(0, 0.5, (H, H))
Wxb = rng.normal(0, 0.5, (X, H)); Whb = rng.normal(0, 0.5, (H, H))

def run(seq, Wx, Wh):
    h = np.zeros(H)
    out = []
    for x in seq:
        h = np.tanh(x @ Wx + h @ Wh)
        out.append(h.copy())
    return np.array(out)

fwd = run(seq, Wxf, Whf)                    # left to right
bwd = run(seq[::-1], Wxb, Whb)[::-1]        # right to left, then re-align
both = np.concatenate([fwd, bwd], axis=1)

print("two SEPARATE RNNs with their own weights, run over the same tokens:")
print("   forward  reads %s" % " -> ".join(words))
print("   backward reads %s" % " -> ".join(words[::-1]))
print()
print("%10s %14s %14s %14s"
      % ("position", "forward h[0]", "backward h[0]", "concatenated"))
for t, w in enumerate(words):
    print("%10s %14.4f %14.4f %14s"
          % (w, fwd[t][0], bwd[t][0], "%d dims" % both.shape[1]))
print()
print("the re-alignment matters and is the usual bug: the backward pass is")
print("computed on the reversed sequence and must be flipped back, so that")
print("position t's forward and backward states describe the SAME token.")
print()

Building on the same setup, the next block shows the left cases.

import numpy as np

print("WHAT EACH DIRECTION KNOWS at position 1 ('bank'):")
print("   forward  has seen: %s" % " ".join(words[:2]))
print("   backward has seen: %s" % " ".join(words[1:][::-1]))
print("   together: the whole sentence.")
print()
print("measure it -- change the LAST word and see which states move:")
alt = seq.copy()
alt[-1] = rng.normal(0, 1.0, X)
f2 = run(alt, Wxf, Whf)
b2 = run(alt[::-1], Wxb, Whb)[::-1]
print("%10s %18s %18s" % ("position", "forward changed?", "backward changed?"))
for t, w in enumerate(words):
    df = np.linalg.norm(fwd[t] - f2[t])
    db = np.linalg.norm(bwd[t] - b2[t])
    print("%10s %18.5f %18.5f" % (w, df, db))
print("   the forward states before the last position are untouched -- they")
print("   were computed before that word arrived. the backward states all")
print("   moved, because for them it was the FIRST word.")
print("   that column is the whole value of the layer.")
print()

print("THE COST. parameters and compute both double:")
uni = X * H + H * H + H
print("   unidirectional : %d parameters, output width %d" % (uni, H))
print("   bidirectional  : %d parameters, output width %d" % (2 * uni, 2 * H))
print("   and the next layer's input is twice as wide, so ITS parameters")
print("   grow too. a stack of bidirectional layers costs more than 2x.")
print()

The below block covers the left steps.

import numpy as np

print("THE RESTRICTION, which is not a detail. the backward pass needs the")
print("whole sequence before it can produce position 0's state:")
print("   generation      -- impossible. you would need the tokens you have")
print("                      not generated yet.")
print("   streaming       -- impossible. no output until the input ends.")
print("   classification  -- fine. you have the whole input already.")
print("   tagging (NER,   -- fine, and this is where it shines: deciding")
print("   POS)               whether 'bank' is a noun or a verb needs the")
print("                      words after it as much as the ones before.")
print()
print("that trade is exactly the BERT/GPT distinction one level down. a")
print("bidirectional encoder and a causal decoder are answering the same")
print("question -- may this position see the future? -- and the answer")
print("decides what the model can be used for.")
print()

print("the practical shape, for a tagging model:")
print("   embed -> BiLSTM -> concat(forward, backward) -> per-token classifier")
print("   output width %d -> %d classes" % (2 * H, 9))
print("   note the classifier sees both halves for every token. that is the")
print("   arrangement, and it was the standard for named-entity recognition")
print("   for most of a decade before encoders replaced it.")

Guided tests

1. Watch both passes. Press Simulate Dual Passes and follow the two chains. One advances left to right, the other right to left. And they meet only where the outputs concatenate.
2. Confirm the doubling. Press Parameters and compare with a unidirectional cell of the same Hidden Size (Per Dir). Just twice as many — two free sets of weights.
3. Check the output width. Set Hidden Size (Per Dir) to 20 and note the layer emits 40 values per spot. The next layer must be built for 40, not 20.
4. Grow the input. Raise Input Vector and watch both directions scale together. The two passes are even in cost; nothing is shared or saved.

When you cannot use it

A bidirectional layer needs the entire sequence before it can give any output, because the backward pass starts at the end. That rules it out when the future truly is not available:

• Language modelling and text generation. Predicting the next token while having already read it is not a prediction. A bidirectional model here reaches perfect precision and learns nothing. The answer leaks in through the backward pass.
• Real-time and streaming uses. Live transcription cannot wait for the end of the utterance.
• Autoregressive decoding of any kind, where output is gave one token at a time.

Where the full sequence is available — classification, tagging, named entity recognition. And the encoder half of a translation model. Bidirectionality is close to free precision. This is just the split between BERT, which is bidirectional and cannot generate, and GPT. This is unidirectional and can.

Traps worth knowing

• Forgetting the output doubles. The most common bug. The below layer is sized for h instead of 2h and the shapes do not match.
• Using it for generation. Training looks superb and the model is useless, because it has been shown the answer.
• Padding without masking. Worse here than in a unidirectional model. The backward pass starts in the padding. So an unmasked model begins reading noise.
• Assuming it doubles quality. It doubles cost. The precision gain is real but often a few points. And on some tasks a wider unidirectional layer is the better use of the same parameters.

The short version

A bidirectional layer runs free forward and backward recurrent passes and concatenates them. So each spot is shown with context from both sides. At twice the parameters and twice the output width. It is the right default for classification and tagging. Where the whole sequence is available, and not doable for generation or streaming, where the backward pass would be reading the future it is meant to predict.

The cost

Twice the work and twice the parameters. Two free LSTMs, each with its own four weight matrices. For 300-dimensional input and 256 hidden units, about 1.15 million parameters instead of 574,000.

Twice the memory during training, since both passes' triggers must be stored.

No latency gain from parallelism. The two directions are free of each other and can run concurrently. But each is still strictly sequential internally.

Full input needed, which is the real cost rather than the arithmetic.

Whether the precision gain justifies it depends on the task. For tagging and span extraction, where the right context is truly needed, the gain is often large. For document-level topic classification, where the sign is distributed and redundant, it is often marginal.

The transformer the same

Bidirectionality in transformers is not a separate layer — it is the default, and unidirectionality is what has to be added.

An encoder's self-attention lets each spot attend to each other, in both directions, with no extra machinery. That is just what BERT is: a bidirectional encoder. And its masked-language-model training objective exists just because bidirectional attention makes next-token prediction trivial.

A decoder adds a causal mask that sets attention scores for future spots to negative infinity. So their weights become zero. Generation becomes doable because the model truly cannot see ahead.

So "bidirectional layer" is largely a recurrent-era concept. The idea survives; the implementation is a mask rather than a second network.

Questions people ask

Can I use a bidirectional layer for language modelling? No. It would see the token it is meant to predict.

Does the backward LSTM share weights with the forward one? No, they are fully separate parameter sets.

Should I concatenate or sum the two directions? Concatenate, which is the standard and keeps both intact. Summing halves the width and loses data.

Is BERT a bidirectional LSTM? No — it is a bidirectional transformer encoder. The word "bidirectional" in its name means unmasked attention.

Does bidirectionality help each task? No. It helps most where right context disambiguates. Tagging, entity recognition, span extraction. And least on tasks with redundant global sign.

Can I stack bidirectional layers? Yes, and each layer's doubled output feeds the next. Two is often the practical limit.

Recap in one screen

• A bidirectional layer runs a second recurrent pass backwards and concatenates the states.
• Each spot then has both left and right context, which resolves ambiguity that neither direction can alone.
• It needs the complete input, so it is unusable for generation and streaming.
• Output width doubles, and the two final states correspond to opposite ends of the sequence.
• In transformers, bidirectionality is the default and a causal mask is what creates the unidirectional case.