The trick is the constraint
An autoencoder is trained to copy its input. Stated like that it's absurd: the identity function scores just right and learns nothing.
The constraint is what makes it a model. Force the network through a form smaller than the input. 64 pixels into 4 numbers — and perfect rebuild turns hopeless. The network has to decide what is worth keeping, and that choice *is* the learned form.
Three parts, and only one of them matters:
- the encoder maps input to a code
z; - the bottleneck is
z itself, whose width you choose; - the decoder maps
z back to a thing the size of the input.
Move the bottleneck slider in the explorer and watch the rebuild. At k = 1 the network has one number per image and all looks like the same mean shape. By k = 4 the six shapes are distinguishable. By k = 16 the rebuild is close, and the compression is only 4×.
Why this page can be exact
A linear autoencoder — no activation functions, just two mats — trained to settling on squared error provably spans the same subspace as the top-k principal components. Baldi and Hornik proved it in 1989: the squared rebuild error has no local minima that aren't global, and each global min spans the principal subspace.
So the explorer doesn't train any of it. It finds the principal components of the dataset right and uses them as the encoder and decoder. There's no seed, no learning rate and no run-to-run variation, and each number on the page is exact.
Two things follow right away, and both are visible.
The error curve is the eigenval tail. Rebuild error at bottleneck k equals the sum of the eigenvals from k+1 onward. That is why the curve in the explorer is smooth and monotone rather than jagged — it isn't a training result, it's math.
The decoder columns are readable. The six small tiles show what each latent dim adds to the rebuild when it's set to +1. Blue is minus. The first accounts for the most variance in the dataset, the second for the most of what is left. So on. The percentages are printed on them.
The latent space is a space
Set the two latent sliders to a coord no real image made and watch the decoder draw a thing anyway.
That is worth pausing on. The decoder is a function defined latent space, not a lookup table of the training set. Feeding it a point between two training images gives a thing between them. Feeding it a point far outside gives a thing, though often not a thing you want.
This is just where a plain autoencoder stops and a variational autoencoder begins. Nothing in the rebuild goal pushes the latent space to be *filled in*. The encoder can scatter the training data into isolated islands with nonsense between them, and the loss won't object — it only ever asks about points that came from real data. A VAE adds a KL term pushing the encoder's output spread toward a standard normal, which pressures the codes to occupy a connected region and makes sampling from the latent space give plausible output. That is the difference between a compressor and a generative model.
Denoising, without asking for it
Turn the corruption slider up. The input tile gets noisy. The rebuild mostly doesn't.
Nothing here was trained to drop noise. The mechanism is the bottleneck: the noise is spread across all 64 directions of pixel space, the bottleneck keeps only k of them, and most of the noise has nowhere to go. The two error figures in the explorer make it concrete. At a narrow bottleneck the rebuild is closer to the *clean* image than the corrupted input was.
A denoising autoencoder makes this the clear goal: corrupt the input, ask for the clean target. It's a stronger idea than it first appears, because it drops the last excuse for learning the identity — the identity is now actively wrong. And it forces the network to learn a thing about the structure of the data rather than about the data itself. It is also, in retrospect, the direct ancestor of masked language modelling and of diffusion models, both of which are "corrupt the input, predict what it was".
Non-linear, and what it buys
If linear autoencoders are PCA, why build a deep one?
Because the principal subspace is a flat subspace, and the structure in real data often isn't flat. Images of a rotating object trace a curve through pixel space. The best 2-plane through that curve is a poor description of it. A non-linear encoder can follow the curve, and a convolutional encoder can also exploit that a shifted image is the same image — a thing a dense linear map has no way to know.
The architecture stays the same three parts. The layer table in the explorer gives a convolutional autoencoder for 28×28 input, and the param counts update with the bottleneck you've selected.
import torch
import torch.nn as nn
class ConvAutoencoder(nn.Module):
def __init__(self, latent=16):
super().__init__()
self.encoder = nn.Sequential(
nn.Conv2d(1, 16, 3, stride=2, padding=1), nn.ReLU(True), # 28 -> 14
nn.Conv2d(16, 32, 3, stride=2, padding=1), nn.ReLU(True), # 14 -> 7
nn.Flatten(),
nn.Linear(32 * 7 * 7, latent))
self.decoder = nn.Sequential(
nn.Linear(latent, 32 * 7 * 7), nn.ReLU(True),
nn.Unflatten(1, (32, 7, 7)),
nn.ConvTranspose2d(32, 16, 3, stride=2, padding=1,
output_padding=1), nn.ReLU(True), # 7 -> 14
nn.ConvTranspose2d(16, 1, 3, stride=2, padding=1,
output_padding=1), # 14 -> 28
nn.Sigmoid())
def forward(self, x):
return self.decoder(self.encoder(x))
model = ConvAutoencoder(latent=16)
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
loss_fn = nn.MSELoss()
for images, _ in loader: # the labels are not used at all
out = model(images)
loss = loss_fn(out, images) # the target IS the input
opt.zero_grad(); loss.backward(); opt.step()
Three details that cause real trouble. output_padding=1 is needed on the transposed convolutions or 7 turns 13 instead of 14 — stride-2 downsampling isn't just invertible and the ambiguity has to be resolved by name. Sigmoid on the output must match the input range: pair it with MSELoss or BCELoss on data in [0, 1], and if your data is normed to [−1, 1] use Tanh instead. And for images, _ in loader is the whole point of the method — the labels are discarded, because the oversight is the input itself.
For the denoising variant, change one line:
noisy = images + 0.3 * torch.randn_like(images)
out = model(noisy.clamp(0, 1))
loss = loss_fn(out, images) # corrupted in, clean out
Choosing the bottleneck
The bottleneck size is the one real setting, and the error curve in the explorer is the tool for choosing it.
The curve is the tail sum of the eigenvals. So Its shape says a thing precise: how much of the dataset's variance is left unexplained after k directions. Where it drops steeply, each new dim is buying a lot. Where it flattens, the left directions are showing noise and per-example detail rather than structure.
The elbow is where to sit. Move the slider on this dataset and the curve falls sharply to about k = 5 and then flattens. Six shapes with a little spot-based and thickness variation truly need about that many numbers. Choosing k = 12 doesn't make the model better. It makes it a slightly lossy copy machine, and the variance-retained figure will read close to 100% while the form has stopped meaning any of it.
Two symptoms tell you which side of the elbow you're on. Too narrow and each rebuild looks like the dataset mean. Watch k = 1 in the explorer. Too wide and the rebuild is great while the latent space is useless for any of it downstream. The encoder has been allowed to pass the input through nearly unchanged rather than show it.
What autoencoders are actually used for
Not compression. JPEG is better, faster and doesn't need a GPU or a training set, and an autoencoder only compresses data resembling what it was trained on.
What they're used for:
- Outlier spotting. Train on normal data only. Any of it the model rebuilds badly is unlike what it saw. This is a truly common production use in making and monitoring, and it works because the bottleneck refuses to represent what it has no basis for.
- The latent space of a diffusion model. Stable Diffusion doesn't diffuse in pixel space. It diffuses in the latent space of a trained autoencoder, at roughly 1/8 the detail per side. The autoencoder is what makes high-detail generation affordable.
- Pretraining and form learning, where masked autoencoders (MAE) brought the idea back for vision transformers. Mask 75% of the patches and rebuild them, which is the denoising goal at an extreme.
The through-line is the same in all three: an autoencoder is a way to find out what a dataset's structure is, by forcing a thing to show it in fewer numbers than it came in.
Questions people ask
If a linear autoencoder is PCA, is it truly PCA? It spans the same *subspace*. This Isn't the same as finding the same axes. Nothing in the goal makes the encoder's directions perp or orders them by variance. A trained linear autoencoder's units are often rotations and mixtures of principal components. This page uses the actual components. This Is why its per-dim tiles are readable and a trained model's wouldn't be.
Why is my autoencoder just copying the input? Because the bottleneck isn't binding. If the code is as wide as the input, or the decoder can simply undo the encoder, the identity is available and it's the optimum. The loss is doing what you asked. Narrow the code, or switch to a denoising goal where the identity is actively wrong.
Autoencoder or VAE? Compressor or generative model. Nothing in the rebuild loss asks the latent space to be filled in. A plain autoencoder is free to scatter codes into isolated islands with nonsense in between. The loss only ever asks about points that came from real data. If you intend to sample from the latent space, you need the KL term that pushes the codes toward a connected region.
Why does the transposed convolution need output_padding=1? Because stride-2 downsampling isn't just invertible. Inputs of 13 and 14 both map to 7. Going back up the ambiguity has to be resolved by name. Leave it out and 7 turns 13 rather than 14. The shape error surfaces a place further down the decoder.
Which output activation and loss go together? Match the data range. Sigmoid with MSELoss or BCELoss for data in [0, 1]. Tanh with MSELoss for data in [−1, 1]. Mismatch them and part of the target sits outside the activation's range. The model spends its room saturating instead of rebuilding.
Is it really any good for compression? No. JPEG is better, faster, needs no GPU and no training set. An autoencoder only compresses data that resembles what it was trained on. The real uses are outlier spotting, supplying the latent space a diffusion model works in, and form pretraining.
Recap in one screen
- The constraint is the model. Forcing the input through a narrower code makes perfect rebuild hopeless. What the network chooses to keep *is* the form.
- A linear autoencoder trained to settling spans the top-k principal subspace. This Is why this page is exact math rather than a training run with a seed.
- The error curve is the eigenval tail. Error at bottleneck k is the sum of eigenvals from k+1 onward. So the elbow is a real thing to read, not a heuristic.
- Denoising falls out of the bottleneck for free. Noise spreads across each direction and only k of them live. Making it the clear goal is the direct ancestor of masked language modelling and of diffusion.
- Non-linearity buys the ability to follow curved structure. This A flat principal subspace shows badly.
- Too narrow and each rebuild is the dataset mean. Too wide and the rebuild is great while the code has stopped meaning any of it.