Natural Language Processing
How text becomes numbers: characters, tokens, embeddings, recurrence and attention.
About this track
Everything a language model does rests on one uncomfortable fact: the model never sees words. This track follows a piece of text all the way down - characters to codes, codes to tokens, tokens to vectors - and then back up through the architectures that read those vectors in order.
It covers the recurrent family in full: what a recurrent cell holds, why backpropagation through time makes long sequences hard, and how LSTM's four gates were designed to fix exactly that. It ends on GRUs and bidirectional layers.
The track follows one piece of text the whole way down, so each stage is visibly the input to the next: characters become tokens, tokens become ids, ids become vectors, and vectors become the thing a model can actually compute with.
What you will be able to do
- Explain why a model never sees words, and what a tokenizer does about it.
- Say what an embedding encodes, and why one-hot vectors cannot encode it.
- Trace a recurrent cell across timesteps and explain what its hidden state holds.
- Describe why long sequences are hard, and how gating in an LSTM addresses it.
- Read self-attention as a weighted lookup rather than as a formula to memorise.
How the track is ordered
Text first: how it is normalised, split and counted, including the n-gram models that preceded neural approaches and still explain why sparsity is the central problem. Then representation - what embeddings are and how they are trained. Then the recurrent family in full, from a bare cell through backpropagation through time to LSTM and bidirectional layers. Attention and positional encoding come last, because they are best understood as answers to the specific limitations the recurrent pages establish. The deep learning track is the useful prerequisite.
Where this leads
Attention is the bridge to the generative AI track, which covers what transformers are trained to do and how they are made small and fast enough to serve. The computer vision track is the parallel specialisation, applying the same base ideas to images instead of sequences.
All 42 modules, in teaching order
- 01ASCII Character CodesType any character and see the ASCII number a computer actually stores, the very first step from text into numbers.
- 02NLP TokenizerSplit text into words, characters or sentences and see how the tokenizer you choose changes what a model actually reads.
- 03Text Normalization PipelineRun raw text through lowercasing, punctuation stripping and stop-word removal, and watch each stage clean it further.
- 04Stemming vs LemmatizationCompare a stemmer chopping suffixes against a lemmatizer resolving proper dictionary forms of the very same words.
- 05N-gram ExplainerSlide an n-gram window across text and see how counting short sequences lets a model guess which word comes next.
- 06Text Classification with Naive BayesType a sentence and watch Naive Bayes score each word's evidence for positive or negative, then classify.
- 07Word Cloud GeneratorPaste any text and watch a word cloud size each term by how often it appears, exposing what a document is really about.
- 08Why Text Encoding is Needed in NLPA neuron computes w*x + b — multiplication needs numbers. See why every NLP pipeline starts by converting words into vectors.
- 09Text Encoding Techniques in NLPClimb from label encoding to one-hot to bag-of-words to TF-IDF, and see which information each rung preserves and which it drops.
- 10How Words are Represented in Neural NetworksTrace the four-stage chain from a raw word to a token ID to a one-hot vector to the dense embedding a network can multiply.
- 11What are Embeddings?Replace sparse one-hot vectors with short dense vectors where similar words sit close together and vector arithmetic encodes meaning.
- 12How are Embeddings Generated?Predict a word from its neighbours with Skip-gram or CBOW and watch the weight matrix converge into vectors where meaning is geometry.
- 13Word2VecTrain skip-gram with negative sampling in the page and watch the nearest neighbour of 'king' become 'queen' as the step count rises.
- 14GloVeBuild the co-occurrence matrix, see the probability ratio that motivates GloVe computed from real counts, and fit the log-bilinear objective.
- 15How Neural Networks Process TextFollow a sentence from raw characters through tokens and vectors into something a neural network can actually multiply.
- 16What is a Sequence?See why 'dog bites man' and 'man bites dog' differ: order carries meaning, and models that ignore it lose the most important signal.
- 17Sequential Data Preparation with Sliding WindowSlide a window across a raw time series to turn one long stream into the (X, y) supervised training pairs a model requires.
- 18Normalization Techniques for Sequential DataSee why batch normalisation breaks on variable-length sequences and how layer normalisation fixes it by computing stats per sample.
- 19Limitations of ANN with Sequential DataSee the three ways a feed-forward network fails on sequences: fixed input size, order blindness, and no memory between steps.
- 20What is a Recurrent Cell?Process one token at a time and pass a fixed-size hidden state forward — the loop that lets a small network read any-length input.
- 21How RNN Processes TextFeed a sentence word by word into a recurrent network and watch the hidden state carry context from each step to the next.
- 22RNN ArchitectureUnroll a recurrent cell across a sequence and see a 50-token sentence become a 50-layer deep network sharing one set of weights.
- 23Interactive 3D RNN VisualizerStep through a recurrent neural network in 3D as it processes a sequence, carrying hidden state from one timestep to the next.
- 24Backpropagation Through TimeUnroll an RNN's loop into T copies and send the gradient backward through each timestep, watching it shrink or explode along the way.
- 25Vanishing Gradient Problem in RNNWatch a gradient shrink by a factor of 0.7 at each timestep until the learning signal at the start of the sentence is near zero.
- 26What is LSTM?Add a cell state that flows through addition instead of matrix multiplication, plus three gates that learn when to remember and forget.
- 27Forget Gate in LSTMSee how a sigmoid between 0 and 1 erases or preserves the cell state, and how forgetting compounds over successive time steps.
- 28Input Gate in LSTMA sigmoid layer that scales the candidate vector element by element, deciding how much new information each memory slot accepts.
- 29Candidate Memory in LSTMThe LSTM layer that manufactures new content: tanh outputs between -1 and +1 for the input gate to accept or reject element by element.
- 30Output Gate in LSTMSee how the output gate filters the cell state through tanh to produce the hidden state that the rest of the network sees.
- 31How LSTM Processes TextWatch an LSTM carry a cell state for long-term memory and a hidden state for short-term as it reads through a sequence.
- 32What is a GRU?Merge the LSTM's three gates into two and its two states into one — same gating idea, 75 percent of the parameters, close accuracy.
- 33What is a Bidirectional Layer?Run one RNN forward and another backward, concatenate their outputs, and give every position context from both sides of the sequence.
- 34Seq2SeqMeasure the seq2seq bottleneck directly: perturb each source token and watch how little of the early sentence survives to the final state.
- 35The Attention MechanismSee why a fixed context vector forgets, and how attention lets the decoder re-weigh every input word at each output step. Full alignment matrix.
- 36Machine Translation with an Encoder-DecoderWatch attention align a French sentence to its English translation, then see greedy decoding drop the negation and beam search recover it.
- 37Query, Key and ValueFollow one attention lookup end to end - score, scale, softmax, blend - and watch softmax saturate when the 1/sqrt(d_k) scaling is removed.
- 38Self-AttentionThe full n x n attention matrix over one sentence. Watch a pronoun resolve at layer 2, apply a causal mask, and see why word order needs encoding.
- 39Multi-Head AttentionRun several attention patterns at once, each tracking a different relationship. See why more heads cost no extra parameters - the dimension is split, not added.
- 40Positional EncodingSelf-attention is blind to word order. See sinusoidal encoding give every position a bounded unique fingerprint, and watch the naive schemes fail.
- 41The Transformer ArchitectureAttention, add, normalise, feed-forward, add, normalise - stacked. Switch the residual connections off and watch a 24-layer stack stop being trainable.
- 42Encoder-only vs Decoder-only (BERT vs GPT)One change to the attention mask splits the transformer family in two. See what each token may look at, and which training objective that permits.