Natural Language Processing
How text becomes numbers: characters, tokens, embeddings, recurrence and attention.
About this track
Everything a language model does rests on one uncomfortable fact: the model never sees words. This track follows a piece of text all the way down - characters to codes, codes to tokens, tokens to vectors - and then back up through the architectures that read those vectors in order.
It covers the recurrent family in full: what a recurrent cell holds, why backpropagation through time makes long sequences hard, and how LSTM's four gates were designed to fix exactly that. It ends on GRUs and bidirectional layers.
The track follows one piece of text the whole way down, so each stage is visibly the input to the next: characters become tokens, tokens become ids, ids become vectors, and vectors become the thing a model can actually compute with.
What you will be able to do
- Explain why a model never sees words, and what a tokenizer does about it.
- Say what an embedding encodes, and why one-hot vectors cannot encode it.
- Trace a recurrent cell across timesteps and explain what its hidden state holds.
- Describe why long sequences are hard, and how gating in an LSTM addresses it.
- Read self-attention as a weighted lookup rather than as a formula to memorise.
How the track is ordered
Text first: how it is normalised, split and counted, including the n-gram models that preceded neural approaches and still explain why sparsity is the central problem. Then representation - what embeddings are and how they are trained. Then the recurrent family in full, from a bare cell through backpropagation through time to LSTM and bidirectional layers. Attention and positional encoding come last, because they are best understood as answers to the specific limitations the recurrent pages establish. The deep learning track is the useful prerequisite.
Where this leads
Attention is the bridge to the generative AI track, which covers what transformers are trained to do and how they are made small and fast enough to serve. The computer vision track is the parallel specialisation, applying the same base ideas to images instead of sequences.
All 41 modules, in teaching order
- 01ASCII Character CodesType any character and see the ASCII number a computer actually stores, the very first step from text into numbers.
- 02NLP TokenizerSplit text into words, characters or sentences and see how the tokenizer you choose changes what a model actually reads.
- 03Text Normalization PipelineRun raw text through lowercasing, punctuation stripping and stop-word removal, and watch each stage clean it further.
- 04Stemming vs LemmatizationCompare a stemmer chopping suffixes against a lemmatizer resolving proper dictionary forms of the very same words.
- 05N-gram ExplainerSlide an n-gram window across text and see how counting short sequences lets a model guess which word comes next.
- 06Word Cloud GeneratorPaste any text and watch a word cloud size each term by how often it appears, exposing what a document is really about.
- 07Why Text Encoding is Needed in NLPFeed raw text into a neuron and watch the math break — then encode it and watch it work. An interactive first-principles demo on VizLearn.
- 08Text Encoding Techniques in NLPCompare Label Encoding, One-Hot, Bag-of-Words and TF-IDF on a live corpus you can edit, side by side, on VizLearn.
- 09How Words are Represented in Neural NetworksFollow a word through the full representation pipeline — token ID, one-hot vector, dense embedding — interactively on VizLearn.
- 10What are Embeddings?Explore a 2D map of word embeddings: click words to measure similarity and run the famous king − man + woman analogy, on VizLearn.
- 11How are Embeddings Generated?Train word embeddings live in your browser: watch random vectors organize themselves into semantic clusters as co-occurring words pull together. On VizLearn.
- 12Word2VecTrain skip-gram with negative sampling in the page and watch the nearest neighbour of 'king' become 'queen' as the step count rises.
- 13GloVeBuild the co-occurrence matrix, see the probability ratio that motivates GloVe computed from real counts, and fit the log-bilinear objective.
- 14How Neural Networks Process TextFollow a sentence from raw characters through tokens and vectors into something a neural network can actually multiply.
- 15What is a Sequence?Understand what sequential data is and why order matters, with an interactive shuffle experiment on VizLearn.
- 16Sequential Data Preparation with Sliding WindowSee how a sliding window turns a raw sequence into supervised training pairs (X, y), step by step, on VizLearn.
- 17Normalization Techniques for Sequential DataCompare Min-Max scaling and Z-Score standardization on a live time series, and see how outliers distort each, on VizLearn.
- 18Limitations of ANN with Sequential DataDiscover why plain feed-forward neural networks struggle with sequences: fixed input size, order blindness, and no memory. Interactive demos on VizLearn.
- 19What is a Recurrent Cell?Step a sentence through a recurrent cell one word at a time and watch the hidden state — the network's memory — evolve. On VizLearn.
- 20How RNN Processes TextFeed a sentence word by word into a recurrent network and watch the hidden state carry context from each step to the next.
- 21RNN ArchitectureLearn what a standard RNN cell is with a beginner-friendly interactive visualization of its memory states and data flow.
- 22Interactive 3D RNN VisualizerSee how a recurrent neural network processes a sequence step by step, carrying hidden state from one time step to the next.
- 23Backpropagation Through TimeUnroll an RNN, run the forward pass, then watch gradients flow backwards through every timestep — with the chain-rule product computed live. On VizLearn.
- 24Vanishing Gradient Problem in RNNPlay with the recurrent weight and sequence length to see gradients vanish or explode — with live decay curves and a log-scale view. On VizLearn.
- 25What is LSTM?Learn what an LSTM cell is with a beginner-friendly interactive visualization of its internal gates, memory states, and data flow.
- 26Forget Gate in LSTMInteractive walkthrough of the Forget Gate in an LSTM cell - see how a sigmoid between 0 and 1 erases or preserves the cell state, and how forgetting compounds over time.
- 27Input Gate in LSTMInteractive walkthrough of the Input Gate in an LSTM cell - see how the sigmoid gate controls which candidate values are written into the cell state.
- 28Candidate Memory in LSTMInteractive walkthrough of the candidate memory layer in an LSTM - see how tanh proposes new signed content for the cell state and why it saturates.
- 29Output Gate in LSTMInteractive walkthrough of the Output Gate in an LSTM cell - see how it filters the cell state through tanh to produce the hidden state that the rest of the network sees.
- 30How LSTM Processes TextWatch an LSTM carry a cell state for long-term memory and a hidden state for short-term as it reads through a sequence.
- 31What is a GRU?Step a sentence through a GRU and watch the update and reset gates blend old memory with new input. Compare parameter counts with LSTM. On VizLearn.
- 32What is a Bidirectional Layer?Learn what a Bidirectional layer is with a beginner-friendly interactive visualization of forward and backward passes.
- 33Seq2SeqMeasure the seq2seq bottleneck directly: perturb each source token and watch how little of the early sentence survives to the final state.
- 34The Attention MechanismSee why a fixed context vector forgets, and how attention lets the decoder re-weigh every input word at each output step. Full alignment matrix.
- 35Machine Translation with an Encoder-DecoderWatch attention align a French sentence to its English translation, then see greedy decoding drop the negation and beam search recover it.
- 36Query, Key and ValueFollow one attention lookup end to end - score, scale, softmax, blend - and watch softmax saturate when the 1/sqrt(d_k) scaling is removed.
- 37Self-AttentionThe full n x n attention matrix over one sentence. Watch a pronoun resolve at layer 2, apply a causal mask, and see why word order needs encoding.
- 38Multi-Head AttentionRun several attention patterns at once, each tracking a different relationship. See why more heads cost no extra parameters - the dimension is split, not added.
- 39Positional EncodingSelf-attention is blind to word order. See sinusoidal encoding give every position a bounded unique fingerprint, and watch the naive schemes fail.
- 40The Transformer ArchitectureAttention, add, normalise, feed-forward, add, normalise - stacked. Switch the residual connections off and watch a 24-layer stack stop being trainable.
- 41Encoder-only vs Decoder-only (BERT vs GPT)One change to the attention mask splits the transformer family in two. See what each token may look at, and which training objective that permits.