Interactive architecture builder. Drag to pan, scroll to zoom, right-click to edit.
Neural networks are powerful math machines, but they only operate on numbers. They can't right understand text like "hello" or "world". The first and most key step in Natural Language Handling (NLP) is to convert text into a numeric format that a network can process. This process is called text vectorization or word embedding. The pipeline has three stages: tokenization splits text into units (words, subwords, or characters), an embedding layer maps each token to a dense vector. And the network processes those vectors through its layers to give an output. A class label for classification, a chance distribution over the vocabulary for generation. Or a vector for likeness search. The embedding layer is the bridge: it turns a discrete symbol (token id 4217) into a smooth vector that the network's arithmetic can operate on. Early approaches used bag-of-words or TF-IDF, which discard word order fully; recurrent networks process tokens sequentially and maintain order through a hidden state; and transformers process all tokens in parallel using self-attention. This is why they dominate current NLP. This page builds a simple text classifier step by step. This shows the tokenized input, the embedding lookup. And the network's forward pass through each layer.
Interactive architecture builder.
Build a neural network and see how it transforms words into numbers to learn, layer by layer.
Neural networks are powerful math machines, but they only operate on numbers. They can't right understand text like "hello" or "world". The first and most key step in Natural Language Handling (NLP) is to convert text into a numeric format that a network can process. This process is called text vectorization or word embedding. The pipeline has three stages: tokenization splits text into units (words, subwords, or characters), an embedding layer maps each token to a dense vector. And the network processes those vectors through its layers to give an output. A class label for classification, a chance distribution over the vocabulary for generation. Or a vector for likeness search. The embedding layer is the bridge: it turns a discrete symbol (token id 4217) into a smooth vector that the network's arithmetic can operate on. Early approaches used bag-of-words or TF-IDF, which discard word order fully; recurrent networks process tokens sequentially and maintain order through a hidden state; and transformers process all tokens in parallel using self-attention. This is why they dominate current NLP. This page builds a simple text classifier step by step. This shows the tokenized input, the embedding lookup. And the network's forward pass through each layer.
Our interactive visualization shows the key stages a neural network follows to process text data:
First, the input sentence is broken down into single units called tokens (often words). Each token is then mapped to a numeric vector called an embedding. This isn't just a random number; it's a multi-dimensional form that captures the word's meaning and context. Words with close meanings will have close embedding vectors.
The numeric embeddings are fed into the hidden layers of the network. Each layer takes the output from the last one and performs complex math transformations. The goal is to combine the initial word-level features into higher-level abstract concepts. For case, the first hidden layer might learn to find pairs of words. While deeper layers might learn to spot sentence structure or sentiment.
Finally, the highly handled data from the last hidden layer reaches the output layer. This layer condenses all the learned features into a final prediction. For a task like sentiment review, the output layer might have two neurons. One for "positive" and one for "negative". The neuron with the highest trigger value decides the model's final answer.
Text does not enter a network as text. Five stages convert it, and each one is a decision that affects what the model can learn.
"the cat sat" → [2, 4756, 3231] → 3 × 768 matrix → context vectors → output
Stage 3 is where the shapes become concrete. A 3-token input with a 768-dimensional embedding is a 3×768 tensor. And that shape — (sequence, features) — is what flows through each later layer.
Models process batches, and sentences differ in length. So short sequences are padded to the batch's longest. And an attention mask marks which spots are real.
| Tokens | Mask | |
|---|---|---|
| "the cat sat" | [2, 4756, 3231, 0, 0] | [1, 1, 1, 0, 0] |
| "a very long sentence here" | [7, 1893, 992, 4410, 651] | [1, 1, 1, 1, 1] |
The mask is not optional. Without it, the model attends to padding as though it were content. And a recurrent model's final hidden state describes the padding rather than the last real word. Results degrade for no visible reason.
Dynamic padding — padding each batch to its own longest sequence rather than to a global maximum — is meaningfully faster when lengths vary. Grouping close lengths into the same batch ("bucketing") reduces the waste further.
| Encoder | How it builds context | Strength |
|---|---|---|
| 1-D CNN | Filters slide along the sequence | Fast; local phrases |
| RNN / LSTM | A carried state, step by step | Bounded memory; streaming |
| Transformer | Attention across all positions | Long range; parallel |
A CNN over text is underrated for classification: a filter of width 3 detects three-word phrases anywhere in the input, it is fast, and it parallelises. Its limit is the receptive field.
An LSTM carries one fixed-size state, which makes it the right choice for streaming and for small datasets.
A transformer makes each spot one step from each other and computes them all at once. This is why it rules at scale.
For most applied work today the practical answer is a pretrained transformer, fine-tuned. Or, with few labels, a frozen encoder plus a small classifier on its embeddings.
Six stages sit between raw text and a number. And each one is a place things go wrong. All six are run here on one sentence, with the shape printed at each boundary.
Building on the same setup, the next block shows the left cases.
The below block covers the left steps.
The next block goes on with the left demonstrations.
Use the network builder above to solidify your grasp:
Neural networks process text by first converting words into meaningful numeric vectors (embeddings). These numbers are then passed through a series of hidden layers that extract increasingly complex features. Finally, an output layer makes a prediction based on these learned features. The entire design, from the input dimensions to the number of hidden layers and neurons, decides the model's capacity to learn and understand language.
The encoder gives one vector per token. What you do with them is determined by the task shape:
| Task | Head |
|---|---|
| Classification | Pool over tokens, then a linear layer to class count |
| Token labelling (NER, POS) | A linear layer applied at every position |
| Span extraction (QA) | Two heads predicting start and end positions |
| Sentence similarity | Pool, normalise, compare with cosine |
| Generation | Linear layer to vocabulary size, sampled repeatedly |
For classification, how you pool matters more than people expect. Mean pooling over the (masked) tokens is a strong default. BERT's [CLS] token works because it was trained for that purpose. Taking the last hidden state of an LSTM over-weights the end of the input.
# masked mean pooling - the reliable default
mask = attention_mask.unsqueeze(-1) # (batch, seq, 1)
summed = (hidden_states * mask).sum(dim=1)
pooled = summed / mask.sum(dim=1).clamp(min=1) # ignore paddingThe clamp guards against an all-padding row, which would if not divide by zero.
Frozen embeddings plus a simple classifier. Embed with a sentence encoder, train logistic regression on the vectors. Minutes of work, no GPU, and with a few hundred labels it often beats fine-tuning because there is far less to overfit.
Fine-tuning a small encoder. A DistilBERT or MiniLM fine-tuned on a few thousand labelled cases. Cheap to run afterwards — milliseconds on a CPU — and often the best precision-per-cost for a fixed task.
Prompting a large model. No training and no labels needed, and much higher inference cost. Right for open-ended tasks or when labels do not exist.
Training from scratch. Almost never the right choice for text. Pretrained forms carry knowledge that a small dataset cannot supply.
And the baseline that should always be measured first: TF-IDF plus logistic regression. On topic classification with adequate labels it is often strong with a fine-tuned transformer, trains in seconds, and is fully interpretable.
Do I need to remove stop words and punctuation? Not for a transformer. They carry syntax and sentiment the model was trained on. Yes, often, for TF-IDF.
How long can the input be? BERT-style models cap at 512 tokens; modern decoders reach far more. Longer documents are chunked, and the chunk results pooled or fetched.
Should I use the [CLS] token or mean pooling? [CLS] for models trained with it; masked mean pooling if not. And it is the safer default.
Why is my model ignoring part of the input? Check truncation. The text may be silently cut at the token limit.
Can I use the same pipeline for a few languages? With a multilingual model, yes. Token counts per word will be higher for languages far from the training distribution.
Do I have to fine-tune? No. Frozen embeddings plus a light classifier is often the better trade with limited labels.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “The Core Challenge: Computers Don't Read Words”?
What does this module say about “The Three Main Stages of Handling Text”?
What does this module say about “Tokenization & Embedding (Input Layer)”?
Follow a sentence from raw characters through tokens and vectors into something a neural network can actually multiply.