Watch the squeeze
The grey line is the input; the orange line is what comes back out.
At a latent size of 1 the reconstruction keeps only the largest structure and misses everything else. Raise it and the copy tightens. Past a certain point extra dimensions buy almost nothing, because the remaining detail is noise rather than signal.
That flattening is the useful observation: it says roughly how many numbers the data really needs, which is its intrinsic dimensionality.
The relationship to PCA
A linear autoencoder trained with squared error finds the same subspace as [PCA](../machine_learning/pca.html). Not something similar — provably the same span, though the individual directions need not be the principal components in order.
So a linear autoencoder is PCA with extra steps. What makes them worth having is non-linearity: with non-linear activations the encoder can learn a curved manifold, which no linear projection can follow. An image dataset lives on a wildly curved surface in pixel space, and that is the case where an autoencoder earns its keep.
Denoising
Raise the noise control. Now the input is corrupted and the target is still the clean signal.
The reconstruction error against the *noisy* input goes up, and against the *clean* signal it goes down. The readout gives both.
The reason is the bottleneck again. Noise is, by construction, the part of the signal with no structure to compress, so a narrow code cannot afford to store it. Forcing the network to reconstruct through the bottleneck forces it to discard exactly what noise is.
Denoising autoencoders make this the training procedure: corrupt the input deliberately, ask for the clean version. It also prevents the degenerate solution where a wide code just learns the identity, which is why the trick appears even when denoising is not the goal.
What they are used for
Dimensionality reduction where the structure is non-linear.
Anomaly detection. Train on normal data; anything that reconstructs badly is unlike what was seen. It works because the model has only learned to compress one kind of thing.
Pretraining. Historically the standard way to initialise deep networks before better initialisation and normalisation made it unnecessary.
Generation — but not directly. A plain autoencoder's latent space has gaps, so a random code usually decodes to nothing meaningful. Fixing that is what [the VAE](variational_autoencoders.html) is for.
Where it goes wrong
A bottleneck that is too wide. With enough capacity the network learns the identity and nothing is discovered. Constrain it, or use denoising, or add a sparsity penalty.
Expecting a usable generative model. Sample from a plain autoencoder's latent space and you will mostly get nonsense.
Reading the reconstruction error as a quality score. It measures pixel agreement, not perceptual similarity, which is why autoencoder outputs look blurred.
Assuming the latent dimensions mean anything individually. They are not ordered or disentangled unless something in the training made them so.