Deep Learning, One Layer at a Time
From a single perceptron to backpropagation, optimizers, regularisation and the shape of a training curve.
About this track
Deep learning is a small number of ideas repeated at scale. This track introduces them in the order they build: one weighted sum, then an activation, then a layer, then a loss, then the gradient that moves the weights, then everything people add to stop it going wrong.
Because each page animates a single step, you can see what a learning rate actually does to a descent path, why a training curve separates from the validation curve, and what dropout removes on each forward pass.
Because every page animates one step rather than a whole system, you can watch a single quantity move - a learning rate stretching a descent path, a validation curve turning upward, a gradient thinning out as it travels back through the layers.
What you will be able to do
- Trace a forward pass by hand: weighted sum, bias, activation, output.
- Say what backpropagation actually computes, and why the chain rule makes depth expensive.
- Diagnose a training run from its curves - underfitting, overfitting, or a broken learning rate.
- Choose an activation, an initialisation and an optimiser, and justify each choice.
- Recognise vanishing and exploding gradients, and name the fixes that address each.
How the track is ordered
The track builds strictly upward. One perceptron first - a weighted sum, a bias, an activation - because everything afterwards is that repeated. Then loss functions, then gradient descent, then backpropagation, which is where the earlier pieces combine. After that come the practical concerns that make deep networks trainable at all: initialisation, normalisation, dropout, regularisation, optimisers and learning-rate schedules. A working knowledge of derivatives helps from the backpropagation module onward; the maths track covers what is needed.
Where this leads
The convolutional and sequence architectures are specialisations of what is here, so computer vision and natural language processing both follow naturally. The generative AI track then picks up where sequence modelling ends, with the training objectives and serving techniques behind current large models.
All 46 modules, in teaching order
- 01Perceptron ClassifierAdjust the weights and bias of a single perceptron by hand and watch its decision boundary swing across the data.
- 02Weights & BiasesSee what the numbers a network learns are actually for: weights scale each input, bias shifts where the neuron fires.
- 03Activation Functions in DLCompare ReLU, sigmoid, tanh and others on the same input, and see how each one bends the signal passing through a layer.
- 04Neural Network VisualizerBuild a neural network layer by layer and watch data flow forward through weights, biases and activations into a prediction.
- 05How Loss is CalculatedFollow a prediction through to a single loss value, and see how the error signal that drives all learning is actually computed.
- 06Softmax and Cross-EntropyDrag raw logits and watch softmax turn them into probabilities, then watch cross-entropy punish a confident mistake. Includes the p - y gradient.
- 07Backpropagation and the Computational GraphStep forward through a small computational graph, then backward, watching each local gradient multiply along the chain. Checked against finite differences.
- 08Gradient DescentWatch gradient descent step downhill across a loss surface, and see how the learning rate decides between converging and diverging.
- 09Linear Regression with Gradient DescentTake gradient descent one step at a time and watch the best-fit line and its mean squared error converge together.
- 10Optimizers in Neural NetworksRun several optimizers in parallel on one loss surface and see how momentum and adaptive learning rates change the route.
- 11Optimizers in 3DRace SGD, Momentum, RMSprop and Adam across a 3D loss landscape and watch which of them escape local minima and which get stuck.
- 12Learning Rate SchedulingCompare a fixed learning rate against a decaying schedule, and watch scheduling rescue a model that would otherwise overshoot.
- 13Batch Processing in NetworksCompare stochastic, mini-batch and full-batch training, and see how batch size trades gradient noise against speed.
- 14Gradient Descent Batch ProcessingChange one control at a time and watch how batch size reshapes the path gradient descent takes toward the minimum.
- 15Feature Scaling & Weight BiasSee what happens when one feature is measured in thousands and another in decimals, and how scaling rebalances the gradients.
- 16Weight Initialization MethodsCompare He, Xavier, random and zero initialisation, and watch poor starting weights kill a deep network before it learns.
- 17Vanishing & Exploding GradientsWatch gradients shrink to nothing or blow up as they multiply back through layers, and see what keeps them in range.
- 18Gradient ClippingStep a weight down a parabola and inject one exploding gradient. Without clipping the weight flies off; with it, the update is capped and training survives.
- 19Residual and Skip ConnectionsChain the same shrinking layer N times and watch the gradient vanish before it reaches the input. Add a skip connection to each layer and watch it stop vanishing.
- 20Batch Normalization in Deep NetworksToggle batch normalisation on a deep network and watch it rescale layer outputs to keep gradients stable as they flow back.
- 21Layer NormalizationNormalize down each sample's own row instead of across the batch, and watch it keep working when the batch shrinks to a single example - where BatchNorm breaks.
- 22Overfitting vs UnderfittingMove model complexity from too simple to too flexible and watch the gap open between training and validation performance.
- 23Model Training CurvesRead training and validation curves like a practitioner: spot underfitting, a healthy fit and overfitting from their shape alone.
- 24Dropout RegularizationSwitch neurons off at random during training and watch dropout stop the network leaning on any single pathway.
- 25Regularization TechniquesCompare L1 and L2 penalties, and watch Lasso zero out useless weights while Ridge shrinks all of them smoothly.
- 26Early StoppingWatch validation loss bottom out then start climbing, and see early stopping halt training at the moment generalisation peaks.
- 27Hyperparameter TuningRun grid search and random search side by side on the same model and see which finds strong hyperparameters in fewer trials.
- 28Data SparsityFeed mostly-zero inputs through a network and watch entire pathways go dormant, because anything multiplied by zero stays zero.
- 29Neural Network for RegressionAdapt a neural network to predict continuous values, and see how its output layer and loss differ from a classifier's.
- 30Neural Network for Unsupervised LearningTrain a network with no labels at all and watch it uncover structure by learning to reconstruct its own input.
- 31Model ReproducibilityFix the random seed and see exactly what becomes repeatable: initial weights and the order training data is shuffled in.
- 32Model Training on CPU vs GPUInteractive visualization of CPU vs GPU in deep learning. Learn how batch size and parallelism affect training speed on VizLearn.
- 33Mixed Precision and Loss Scaling
- 34Gradient Accumulation
- 35Label Smoothing
- 36Embedding Layers
- 37Autoencoders
- 38Autoencoders in DepthMove an autoencoder's bottleneck and watch the reconstruction, the learned components and the exact error curve respond - no training run, no seed.
- 39Variational Autoencoders
- 40Generative Adversarial Networks
- 41GAN Architecture and EquilibriumBe the generator: move the distribution and watch the exact optimal discriminator, the -log 4 equilibrium and the vanishing gradient respond.
- 42Diffusion Models
- 43Contrastive and Self-Supervised Learning
- 44Seq2seq and Beam Search
- 45Collaborative FilteringClick any cell of a ratings matrix and watch user-kNN, item-kNN and matrix factorisation predict it, all scored on the same held-out ratings.
- 46Deep Learning for Recommendation SystemsTwo towers, an embedding table and an MLP: see why 99% of a neural recommender is lookup tables and what deep learning actually buys.