Build and understand deep learning architectures designed to find hidden patterns in data without human labels.
An autoencoder is trained to reproduce its own input. On its own that is trivial. Copy the input to the output and the loss is zero. The trick is the bottleneck: a hidden layer narrower than the input. It the data must pass through.
Since the bottleneck cannot carry all, the network is forced to decide what matters. What survives the squeeze is a learned, compressed form. And it is learned without a single label. The encoder maps the input down to the bottleneck. The decoder maps the bottleneck back up to the first dimension. The loss is the rebuild error (often MSE). And backpropagation trains both halves jointly. After training, the encoder alone is useful: it gives a low-dim form of any input that gets the structure the decoder needed to reconstruct it. This is dim count reduction, like PCA. But nonlinear — the hidden layers can learn curved manifolds that a linear way would miss. Variants include denoising autoencoders (corrupt the input, reconstruct the clean version, which forces the bottleneck to learn noise-resistant features), variational autoencoders (VAEs, which learn a chance spread over the latent space and can generate new samples). And sparse autoencoders (which penalise output to learn a wider but mostly-inactive bottleneck). This page trains a small autoencoder on 8-dim data, prints the rebuild error. And shows how the 3-dim bottleneck gets the dominant correlations.
Build and understand deep learning architectures designed to find hidden patterns in data without human labels.
With no labels to learn from, you make the input its own target. Squeeze it through a narrow layer and the network has to work out what was worth keeping.
An autoencoder is trained to reproduce its own input. On its own that is trivial. Copy the input to the output and the loss is zero. The trick is the bottleneck: a hidden layer narrower than the input. It the data must pass through.
Since the bottleneck cannot carry all, the network is forced to decide what matters. What survives the squeeze is a learned, compressed form. And it is learned without a single label. The encoder maps the input down to the bottleneck. The decoder maps the bottleneck back up to the first dimension. The loss is the rebuild error (often MSE). And backpropagation trains both halves jointly. After training, the encoder alone is useful: it gives a low-dim form of any input that gets the structure the decoder needed to reconstruct it. This is dim count reduction, like PCA. But nonlinear — the hidden layers can learn curved manifolds that a linear way would miss. Variants include denoising autoencoders (corrupt the input, reconstruct the clean version, which forces the bottleneck to learn noise-resistant features), variational autoencoders (VAEs, which learn a chance spread over the latent space and can generate new samples). And sparse autoencoders (which penalise output to learn a wider but mostly-inactive bottleneck). This page trains a small autoencoder on 8-dim data, prints the rebuild error. And shows how the 3-dim bottleneck gets the dominant correlations.
8 input features → 3-unit bottleneck → 8 reconstruction outputs. The encoder must express 8 counts with 3. And the decoder must rebuild all 8 from those 3.
If those 8 features are genuinely free, the reconstruction will be poor. There is nothing to compress. If a few are linked, as real features often are, the network finds those links and reconstruction gets surprisingly close. The loss is simply MSE(input, reconstruction).
Supervised learning needs a target for each case. Most data has none. And labelling it is the costly part of each project.
Unsupervised ways learn structure from the data it. For neural networks the dominant trick is to invent a target from the input. So that ordinary supervised machinery can be used on unlabelled data. That reframing is called self-supervision. And it is what made modern language and vision pretraining possible.
The main families:
| Approach | Invented target | Used for |
|---|---|---|
| Autoencoder | Reconstruct the input | Compression, denoising, anomaly detection |
| Masked prediction | Predict hidden parts | BERT, MAE — pretraining |
| Next-token prediction | Predict what follows | Language models |
| Contrastive | Are these two views the same item? | SimCLR, CLIP — representation learning |
| Generative (VAE, GAN, diffusion) | Produce plausible samples | Image and audio generation |
An autoencoder is two networks back to back. The encoder compresses the input into a small vec. The decoder reconstructs the input from it. The loss is rebuild error. How far the output is from the input.
x → encoder → z (small) → decoder → x̂ loss = ‖x̂. − x‖²
The bottleneck is the whole design. Force 784 pixels through a 32-dim vec and the network cannot memorise. It must find the structure that lets it rebuild the input from 32 counts. Those 32 counts are a learned compression of the data.
Remove the bottleneck — make z as large as the input. And the network learns the identity function. It teaches it nothing. Constraint is what creates the form.
Three real uses:
Anomaly detection. Train on normal data just. Anything that reconstructs badly is unlike what the net has seen. And rebuild error turns an anomaly score. This is one of the most widely deployed unsupervised tricks in industry.
Denoising. Feed a corrupted input and ask for the clean first. The network cannot copy its input. So it must learn what the data ought to look like.
Dim count reduction. A non-linear option to PCA. With linear outputs and squared error, an autoencoder recovers the PCA subspace just. So it is a strict transfer.
The idea: take two tweaked views of the same pic. And train the network to give close embeddings for them and dissimilar embeddings for views of other pics.
No labels are needed — the supervision comes fully from knowing which pairs came from the same source. What the network learns is what is invariant under the augmentations you chose: crop it, recolour it, flip it. And it is still the same object.
That last point is the crux. The augmentations set what the form considers key. Include colour jitter and the net learns colour-invariant features. It is right for object recognition and wrong for distinguishing ripe from unripe fruit.
Contrastive pretraining on unlabelled data followed by fine-tuning on a small labelled set routinely beats training from scratch on the labelled set alone. And in specialised domains it beats ImageNet setup too. CLIP extends the idea across modalities, matching pics with their captions. It is what gives it zero-shot sorting.
A bottleneck as wide as the input. The network learns the identity function, the loss looks excellent. And you have learned nothing about the data. If reconstruction is perfect, suspect this first.Reading reconstruction loss as quality. Low loss just means the input was reproducible. It says nothing about whether the compressed form is useful for anything else. Judge it by how well the bottleneck features do on a downstream task.
No labels needed — make the input the target and let a narrow layer decide what was worth keeping.
A plain autoencoder's latent space has gaps: pick a random point in it and the decoder gives nonsense, since nothing constrained the space to be continuous.
A variational autoencoder fixes that by having the encoder output a spread rather than a point. And adding a KL divergence term that pulls those spreads towards a standard normal. The latent space turns smooth and samplable. So you can generate new cases by drawing a random point and decoding it.
The trade is sharpness: VAE samples tend to be blurry, since the reconstruction loss rewards the average of plausible outputs.
GANs take a other route. A generator and a discriminator trained against each other. Giving sharper samples at the cost of unstable training and mode collapse. Diffusion nets learn to reverse a gradual noising process. And now dominate pic generation on quality and stability alike.
All three are unsupervised in the same sense: the training signal comes from the data, not from labels.
Classical clustering — k-means, DBSCAN, hierarchical. Operates on the raw features and struggles on high-dim unstructured data, since distances there are not meaningful.
The standard modern pipeline is two stages: learn a form with a self-supervised network, then cluster in that embedding space with an ordinary algorithm. Pics that look alike land near each other in the embedding. So k-means on embeddings finds groups that k-means on pixels never could.
Deep clustering ways (DEC and successors) train the form and the clustering jointly. It can work better and is much more fiddly.
Reconstruction, clustering, self-prediction and density -- four objectives that need no target column, run on the same data so you can see what each one really tunes.
Building on the same setup, the next block shows the left cases.
The next block covers the left steps.
The next block continues with the left demonstrations.
Building on the same setup, the next block shows the left cases.
Is self-supervised learning the same as unsupervised? Self-supervision is a subset: it invents labels from the data. The training loop is then fully supervised.
How small should an autoencoder's bottleneck be? Small enough to force compression. Treat it as a hyperparameter. And watch reconstruction quality against the downstream task.
Are autoencoders good for anomaly detection? Yes. And with a caveat: train on normal data just. And be aware that a sufficiently flexible autoencoder can learn to reconstruct anomalies too.
Should I use PCA or an autoencoder? PCA first. It is faster, deterministic and readable. An autoencoder wins when the structure is genuinely non-linear and there is enough data.
Do I still need labels? For the final task, often a few. The point of pretraining is that a small labelled set turns enough.
Which self-supervised way should I use? For pics, a contrastive or masked-pic way. For text, masked or next-token prediction. Or start from a published pretrained net rather than doing it yourself.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
What does this module say about “The autoencoder trick”?
What does this module say about “A worked shape”?
Train a network with no labels at all and watch it uncover structure by learning to reconstruct its own input.