Build and understand deep learning architectures designed to find hidden patterns in data without human labels.
An autoencoder is trained to reproduce its own input. On its own that is trivial — copy the input to the output and the loss is zero. The trick is the bottleneck: a hidden layer narrower than the input, which the data must pass through.
Because the bottleneck cannot carry everything, the network is forced to decide what matters. What survives the squeeze is a learned, compressed representation, and it is learned without a single label.
With no labels to learn from, you make the input its own target. Squeeze it through a narrow layer and the network has to work out what was worth keeping.
8 input features → 3-unit bottleneck → 8 reconstruction outputs. The encoder must express 8 numbers using 3, and the decoder must rebuild all 8 from those 3.
If those 8 features are genuinely independent, the reconstruction will be poor — there is nothing to compress. If several are correlated, as real features usually are, the network finds those relationships and reconstruction gets surprisingly close. The loss is simply MSE(input, reconstruction).
Supervised learning needs a target for every example. Most data has none — and labelling it is the expensive part of every project.
Unsupervised methods learn structure from the data itself. For neural networks the dominant trick is to invent a target from the input, so that ordinary supervised machinery can be used on unlabelled data. That reframing is called self-supervision, and it is what made modern language and vision pretraining possible.
The main families:
| Approach | Invented target | Used for |
|---|---|---|
| Autoencoder | Reconstruct the input | Compression, denoising, anomaly detection |
| Masked prediction | Predict hidden parts | BERT, MAE — pretraining |
| Next-token prediction | Predict what follows | Language models |
| Contrastive | Are these two views the same item? | SimCLR, CLIP — representation learning |
| Generative (VAE, GAN, diffusion) | Produce plausible samples | Image and audio generation |
An autoencoder is two networks back to back. The encoder compresses the input into a small vector; the decoder reconstructs the input from it. The loss is reconstruction error — how different the output is from the input.
x → encoder → z (small) → decoder → x̂ loss = ‖x̂ − x‖²
The bottleneck is the whole design. Force 784 pixels through a 32-dimensional vector and the network cannot memorise; it must find the structure that lets it rebuild the input from 32 numbers. Those 32 numbers are a learned compression of the data.
Remove the bottleneck — make z as large as the input — and the network learns the identity function, which teaches it nothing. Constraint is what creates the representation.
Three practical uses:
Anomaly detection. Train on normal data only. Anything that reconstructs badly is unlike what the model has seen, and reconstruction error becomes an anomaly score. This is one of the most widely deployed unsupervised techniques in industry.
Denoising. Feed a corrupted input and ask for the clean original. The network cannot copy its input, so it must learn what the data should look like.
Dimensionality reduction. A non-linear alternative to PCA. With linear activations and squared error, an autoencoder recovers the PCA subspace exactly — so it is a strict generalisation.
The idea: take two augmented views of the same image, and train the network to produce similar embeddings for them and dissimilar embeddings for views of different images.
No labels are needed — the supervision comes entirely from knowing which pairs came from the same source. What the network learns is what is invariant under the augmentations you chose: crop it, recolour it, flip it, and it is still the same object.
That last point is the crux. The augmentations define what the representation considers important. Include colour jitter and the model learns colour-invariant features, which is right for object recognition and wrong for distinguishing ripe from unripe fruit.
Contrastive pretraining on unlabelled data followed by fine-tuning on a small labelled set routinely beats training from scratch on the labelled set alone, and in specialised domains it beats ImageNet initialisation too. CLIP extends the idea across modalities, matching images with their captions, which is what gives it zero-shot classification.
A bottleneck as wide as the input. The network learns the identity function, the loss looks excellent, and you have learned nothing about the data. If reconstruction is perfect, suspect this first.Reading reconstruction loss as quality. Low loss only means the input was reproducible, which says nothing about whether the compressed representation is useful for anything else. Judge it by how well the bottleneck features perform on a downstream task.
No labels needed — make the input the target and let a narrow layer decide what was worth keeping.
A plain autoencoder's latent space has gaps: pick a random point in it and the decoder produces nonsense, because nothing constrained the space to be continuous.
A variational autoencoder fixes that by having the encoder output a distribution rather than a point, and adding a KL divergence term that pulls those distributions towards a standard normal. The latent space becomes smooth and samplable, so you can generate new examples by drawing a random point and decoding it.
The trade is sharpness: VAE samples tend to be blurry, because the reconstruction loss rewards the average of plausible outputs.
GANs take a different route — a generator and a discriminator trained against each other — producing sharper samples at the cost of unstable training and mode collapse. Diffusion models learn to reverse a gradual noising process, and now dominate image generation on quality and stability alike.
All three are unsupervised in the same sense: the training signal comes from the data, not from labels.
Classical clustering — k-means, DBSCAN, hierarchical — operates on the raw features and struggles on high-dimensional unstructured data, because distances there are not meaningful.
The standard modern pipeline is two stages: learn a representation with a self-supervised network, then cluster in that embedding space with an ordinary algorithm. Images that look alike land near each other in the embedding, so k-means on embeddings finds groups that k-means on pixels never could.
Deep clustering methods (DEC and successors) train the representation and the clustering jointly, which can work better and is considerably more fiddly.
Reconstruction, clustering, self-prediction and density -- four objectives that need no target column, run on the same data so you can see what each one actually optimises.
Is self-supervised learning the same as unsupervised? Self-supervision is a subset: it invents labels from the data. The training loop is then fully supervised.
How small should an autoencoder's bottleneck be? Small enough to force compression. Treat it as a hyperparameter, and watch reconstruction quality against the downstream task.
Are autoencoders good for anomaly detection? Yes, and with a caveat: train on normal data only, and be aware that a sufficiently flexible autoencoder can learn to reconstruct anomalies too.
Should I use PCA or an autoencoder? PCA first — it is faster, deterministic and interpretable. An autoencoder wins when the structure is genuinely non-linear and there is enough data.
Do I still need labels? For the final task, usually a few. The point of pretraining is that a small labelled set becomes sufficient.
Which self-supervised method should I use? For images, a contrastive or masked-image approach; for text, masked or next-token prediction. Or start from a published pretrained model rather than doing it yourself.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
No labels needed — make the input the target and let a narrow layer decide what was worth keeping.
What does this module say about “The autoencoder trick”?
An autoencoder is trained to reproduce its own input. On its own that is trivial — copy the input to the output and the loss is zero. The trick is the bottleneck: a hidden layer narrower than the input, which the data must pass through.
What does this module say about “A worked shape”?
8 input features → 3-unit bottleneck → 8 reconstruction outputs. The encoder must express 8 numbers using 3, and the decoder must rebuild all 8 from those 3.
An autoencoder is trained to reproduce its own input. On its own that is trivial — copy the input to the output and the loss is zero. The trick is the bottleneck: a hidden layer narrower than the input, which the data must pass through.