Interactive architecture builder. Drag to pan, scroll to zoom, right-click to edit.
Neural networks are powerful mathematical machines, but they only operate on numbers. They can't directly understand text like "hello" or "world". The first and most crucial step in Natural Language Processing (NLP) is to convert text into a numerical format that a network can process. This process is called text vectorization or word embedding.
Build a neural network and see how it transforms words into numbers to learn, layer by layer.
Our interactive visualization demonstrates the key stages a neural network follows to process text data:
First, the input sentence is broken down into individual units called tokens (usually words). Each token is then mapped to a numerical vector called an embedding. This isn't just a random number; it's a multi-dimensional representation that captures the word's meaning and context. Words with similar meanings will have similar embedding vectors.
The numerical embeddings are fed into the hidden layers of the network. Each layer takes the output from the previous one and performs complex mathematical transformations. The goal is to combine the initial word-level features into higher-level abstract concepts. For example, the first hidden layer might learn to identify pairs of words, while deeper layers might learn to recognize sentence structure or sentiment.
Finally, the highly processed information from the last hidden layer reaches the output layer. This layer condenses all the learned features into a final prediction. For a task like sentiment analysis, the output layer might have two neurons: one for "positive" and one for "negative". The neuron with the highest activation value determines the model's final answer.
Text does not enter a network as text. Five stages convert it, and each one is a decision that affects what the model can learn.
"the cat sat" → [2, 4756, 3231] → 3 × 768 matrix → contextual vectors → output
Stage 3 is where the shapes become concrete. A 3-token input with a 768-dimensional embedding is a 3×768 tensor, and that shape — (sequence, features) — is what flows through every subsequent layer.
Models process batches, and sentences differ in length. So short sequences are padded to the batch's longest, and an attention mask marks which positions are real.
| Tokens | Mask | |
|---|---|---|
| "the cat sat" | [2, 4756, 3231, 0, 0] | [1, 1, 1, 0, 0] |
| "a very long sentence here" | [7, 1893, 992, 4410, 651] | [1, 1, 1, 1, 1] |
The mask is not optional. Without it, the model attends to padding as though it were content, and a recurrent model's final hidden state describes the padding rather than the last real word. Results degrade for no visible reason.
Dynamic padding — padding each batch to its own longest sequence rather than to a global maximum — is meaningfully faster when lengths vary. Grouping similar lengths into the same batch ("bucketing") reduces the waste further.
| Encoder | How it builds context | Strength |
|---|---|---|
| 1-D CNN | Filters slide along the sequence | Fast; local phrases |
| RNN / LSTM | A carried state, step by step | Bounded memory; streaming |
| Transformer | Attention across all positions | Long range; parallel |
A CNN over text is underrated for classification: a filter of width 3 detects three-word phrases anywhere in the input, it is fast, and it parallelises. Its limit is the receptive field.
An LSTM carries one fixed-size state, which makes it the right choice for streaming and for small datasets.
A transformer makes every position one step from every other and computes them all at once, which is why it dominates at scale.
For most applied work today the practical answer is a pretrained transformer, fine-tuned — or, with few labels, a frozen encoder plus a small classifier on its embeddings.
Six stages sit between raw text and a number, and each one is a place things go wrong. All six are run here on one sentence, with the shape printed at every boundary.
Use the network builder above to solidify your understanding:
Neural networks process text by first converting words into meaningful numerical vectors (embeddings). These numbers are then passed through a series of hidden layers that extract increasingly complex features. Finally, an output layer makes a prediction based on these learned features. The entire architecture, from the input dimensions to the number of hidden layers and neurons, determines the model's capacity to learn and understand language.
The encoder produces one vector per token. What you do with them is determined by the task shape:
| Task | Head |
|---|---|
| Classification | Pool over tokens, then a linear layer to class count |
| Token labelling (NER, POS) | A linear layer applied at every position |
| Span extraction (QA) | Two heads predicting start and end positions |
| Sentence similarity | Pool, normalise, compare with cosine |
| Generation | Linear layer to vocabulary size, sampled repeatedly |
For classification, how you pool matters more than people expect. Mean pooling over the (masked) tokens is a strong default. BERT's [CLS] token works because it was trained for that purpose. Taking the last hidden state of an LSTM over-weights the end of the input.
# masked mean pooling - the reliable default
mask = attention_mask.unsqueeze(-1) # (batch, seq, 1)
summed = (hidden_states * mask).sum(dim=1)
pooled = summed / mask.sum(dim=1).clamp(min=1) # ignore paddingThe clamp guards against an all-padding row, which would otherwise divide by zero.
Frozen embeddings plus a simple classifier. Embed with a sentence encoder, train logistic regression on the vectors. Minutes of work, no GPU, and with a few hundred labels it frequently beats fine-tuning because there is far less to overfit.
Fine-tuning a small encoder. A DistilBERT or MiniLM fine-tuned on a few thousand labelled examples. Cheap to run afterwards — milliseconds on a CPU — and usually the best accuracy-per-cost for a fixed task.
Prompting a large model. No training and no labels needed, and much higher inference cost. Right for open-ended tasks or when labels do not exist.
Training from scratch. Almost never the right choice for text. Pretrained representations carry knowledge that a small dataset cannot supply.
And the baseline that should always be measured first: TF-IDF plus logistic regression. On topic classification with adequate labels it is frequently competitive with a fine-tuned transformer, trains in seconds, and is fully interpretable.
Do I need to remove stop words and punctuation? Not for a transformer — they carry syntax and sentiment the model was trained on. Yes, often, for TF-IDF.
How long can the input be? BERT-style models cap at 512 tokens; modern decoders reach far more. Longer documents are chunked, and the chunk results pooled or retrieved.
Should I use the [CLS] token or mean pooling? [CLS] for models trained with it; masked mean pooling otherwise, and it is the safer default.
Why is my model ignoring part of the input? Check truncation — the text may be silently cut at the token limit.
Can I use the same pipeline for several languages? With a multilingual model, yes. Token counts per word will be higher for languages far from the training distribution.
Do I have to fine-tune? No. Frozen embeddings plus a light classifier is often the better trade with limited labels.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
Neural networks process text by first converting words into meaningful numerical vectors (embeddings). These numbers are then passed through a series of hidden layers that extract increasingly complex features. Finally, an output layer makes a prediction based on these learned features.
What does this module say about “The Core Challenge: Computers Don't Read Words”?
Neural networks are powerful mathematical machines, but they only operate on numbers. They can't directly understand text like "hello" or "world". The first and most crucial step in Natural Language Processing (NLP) is to convert text into a numerical format that a network can process. This process is called text vectorization or word embedding .
What does this module say about “The Three Main Stages of Processing Text”?
Our interactive visualization demonstrates the key stages a neural network follows to process text data:
Neural networks are powerful mathematical machines, but they only operate on numbers. They can't directly understand text like "hello" or "world". The first and most crucial step in Natural Language Processing (NLP) is to convert text into a numerical format that a network can process. This process is called text vectorization or word embedding.