Force a signal through a narrow gap and out again. What survives is what the network decided mattered.
Overview
The apparently useless task
An autoencoder has two halves. The encoder maps an input to a smaller form. The decoder maps it back. The loss is the difference between the output and the input.
Copy the input to the output. Stated like that it sounds like a null task — the identity function scores just right.
Except it can't, because the form in the middle is smaller than the input. The network has to throw info away, and the training goal forces it to throw away the least costly info. What lives the bottleneck is what the data was mostly made of.
Autoencoders
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
An autoencoder learns to copy its input. The copying is not the point — the bottleneck is.
With fewer latent dimensions than inputs, an exact copy is impossible, so the network must decide what to keep.
A linear autoencoder with squared error provably finds the same subspace as PCA.
Add noise to the input and ask for the clean output: the bottleneck cannot afford to store noise, so it removes it.
Autoencoders
A network trained to copy its input, and why that odd task is handy.
Watch the squeeze
The grey line is the input. The orange line is what comes back out.
At a latent size of 1 the rebuild keeps only the largest structure and misses all else. Raise it and the copy tightens. Past a certain point extra dims buy almost nothing, because the left detail is noise rather than signal.
That flattening is the handy reading: it says roughly how many numbers the data really needs, which is its true dimensionality.
The relationship to PCA
A linear autoencoder trained with squared error finds the same subspace as PCA. Not a thing similar — provably the same span, though the single directions need not be the principal components in order.
So a linear autoencoder is PCA with extra steps. What makes them worth having is non-linearity: with non-linear activations the encoder can learn a curved manifold, which no linear map can follow. An image dataset lives on a wildly curved surface in pixel space, and that is the case where an autoencoder earns its keep.
Denoising
Raise the noise control. Now the input is corrupted and the target is still the clean signal.
The rebuild error against the *noisy* input goes up, and against the *clean* signal it goes down. The readout gives both.
The reason is the bottleneck again. Noise is, by build, the part of the signal with no structure to compress. So A narrow code can't afford to store it. Forcing the network to rebuild through the bottleneck forces it to discard just what noise is.
Denoising autoencoders make this the training method: corrupt the input on purpose, ask for the clean version. It also blocks the collapse answer where a wide code just learns the identity. This Is why the trick appears even when denoising isn't the goal.
What they are used for
Dimensionality reduction where the structure is non-linear.
Outlier spotting. Train on normal data. Any of it that rebuilds badly is unlike what was seen. It works because the model has only learned to compress one kind of thing.
Pretraining. Once the standard way to reset deep networks before better init and norming made it unneeded.
Generation — but not right. A plain autoencoder's latent space has gaps, so a random code often decodes to nothing real. Fixing that is what the VAE is for.
The elbow that reveals intrinsic dimension
The handy thing an autoencoder measures is *how many numbers the data really needs*, and it shows up as an elbow in rebuild error against latent size. On data that truly lives in d dims with a little noise on top, widening the bottleneck past d buys almost nothing:
latent size
reconstruction error
1
large — only the biggest structure survives
up to the true d
falling steeply, each dimension earns its place
beyond d
flat — the remaining detail is noise
The point where the curve flattens is the data's true dimensionality, and it is the same number PCA would find — a linear autoencoder with squared-error loss provably recovers the PCA subspace. The reason to reach for an autoencoder instead is the non-linear case: real data (an image dataset, say) lies on a curved manifold in a high-dimensional space, and a straight map can't follow the curve where a non-linear encoder can. That is also why the rebuild error reads as pixel match rather than visual quality, and why autoencoder outputs look softened: the bottleneck keeps what is common to the data and spends nothing on the exact sharp detail that sets apart one example from its neighbours.
The denoising variant turns the same bottleneck into a training method: corrupt the input on purpose and ask for the clean version back. Because noise is by build the part of a signal with no structure to compress, a narrow code can't afford to store it. So Rebuilding through the bottleneck forces the network to discard just what noise is. That also drops the collapse escape where a too-wide code simply learns the identity. This Is why the trick shows up even when denoising isn't the goal. And it's the same instinct behind using an autoencoder for outlier spotting. Here A model trained only on normal data rebuilds any of it unseen badly, and the rebuild error turns the outlier score.
None of the latent dims mean any of it on their own, either: they aren't sorted or disentangled unless a thing in the training made them so, which is a large part of what separates a plain autoencoder from the variational and beta-weighted variants built to push just that structure.
Where it goes wrong with Autoencoders
A bottleneck that is too wide. With enough room the network learns the identity and nothing is found. Constrain it, or use denoising, or add a sparsity fine.
Expecting a usable generative model. Sample from a plain autoencoder's latent space and you'll mostly get nonsense.
Reading the rebuild error as a quality score. It measures pixel match, not visual likeness. This Is why autoencoder outputs look blurred.
given the latent dims mean any of it on their own. They aren't sorted or disentangled unless a thing in the training made them so.
Questions people ask about Autoencoders
What is an autoencoder? A network trained to reproduce its input after squeezing it through a narrow bottleneck. The bottleneck forces it to learn a compact code.
How is an autoencoder related to PCA? A linear autoencoder with squared error learns the same subspace as PCA. Non-linear activations let it capture curved structure that PCA cannot.
Why not just make the bottleneck as big as the input? Then the network can learn the identity function and copy the input without learning anything useful. A small bottleneck, noise or a sparsity penalty prevents that.
What is a denoising autoencoder? One trained to recover the clean input from a corrupted copy. This pushes it to learn the structure of the data rather than memorise it.
What are autoencoders used for? Dimensionality reduction, anomaly detection through high reconstruction error, pretraining and compression. They are not good generators by themselves.
How do I choose the bottleneck size? Plot reconstruction error against code size and look for the elbow. Beyond the elbow, extra dimensions add little.
Train one, then look at what the bottleneck kept
An autoencoder learns to copy its own input through a narrow layer. Training one on data with a known hidden structure shows what a bottleneck forces it to discover -- and where PCA does the job better.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
N, OBS = 300, 12
def standardise(a): return (a - a.mean(0)) / a.std(0)
factors = rng.normal(size=(N, 2))
mix = rng.normal(size=(2, OBS))
X = standardise(factors @ mix + rng.normal(0, 0.05, (N, OBS)))
def ae(X, k, ep=800, lr=0.04):
r=np.random.default_rng(0); h=12; w=lambda a,b: r.normal(0,.3,(a,b))
P=[w(OBS,h),np.zeros((1,h)),w(h,k),np.zeros((1,k)),
w(k,h),np.zeros((1,h)),w(h,OBS),np.zeros((1,OBS))]
m=[np.zeros_like(p) for p in P]; v=[np.zeros_like(p) for p in P]
for t in range(1, ep+1):
a1=np.tanh(X@P[0]+P[1]); cd=a1@P[2]+P[3]
a2=np.tanh(cd@P[4]+P[5]); out=a2@P[6]+P[7]; d=2*(out-X)/len(X)
g6,g7=a2.T@d, d.sum(0,keepdims=True)
d2=(d@P[6].T)*(1-a2**2); g4,g5=cd.T@d2, d2.sum(0,keepdims=True)
dc=d2@P[4].T; g2,g3=a1.T@dc, dc.sum(0,keepdims=True)
d1=(dc@P[2].T)*(1-a1**2); g0,g1=X.T@d1, d1.sum(0,keepdims=True)
for i,g in enumerate((g0,g1,g2,g3,g4,g5,g6,g7)):
m[i]=.9*m[i]+.1*g; v[i]=.999*v[i]+.001*g**2
P[i]-=lr*(m[i]/(1-.9**t))/(np.sqrt(v[i]/(1-.999**t))+1e-8)
a1=np.tanh(X@P[0]+P[1]); cd=a1@P[2]+P[3]
return cd, ((np.tanh(cd@P[4]+P[5])@P[6]+P[7]-X)**2).mean()
print("%d rows, %d observed columns, secretly built from 2 hidden factors." % (N, OBS))
print("the autoencoder is never told that number.")
print("reconstruction error against the size of the bottleneck:")
print("%12s %18s %14s" % ("bottleneck", "reconstruction", "compression"))
errs = {}
for k in (1, 2, 3, 6):
code, mse = ae(X, k)
errs[k] = (code, mse)
print("%12d %18.6f %13.1fx" % (k, mse, OBS / k))
drop_1_2 = errs[1][1] / errs[2][1]
drop_2_3 = errs[2][1] / errs[3][1]
print("going from 1 to 2 divides error by %.0f. 2 to 3 changes it by %.2f." % (drop_1_2, drop_2_3))
print("that knee tells you the data is 2-dimensional, which it is.")
print("this is how autoencoders estimate intrinsic dimension.")
Output
The sharp drop from one unit to two is the model discovering the true dimensionality. Now look at what the code layer actually contains.
example_02.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
N, OBS = 300, 12
def standardise(a): return (a - a.mean(0)) / a.std(0)
factors = rng.normal(size=(N, 2))
mix = rng.normal(size=(2, OBS))
X = standardise(factors @ mix + rng.normal(0, 0.05, (N, OBS)))
def ae(X, k, ep=800, lr=0.04):
r=np.random.default_rng(0); h=12; w=lambda a,b: r.normal(0,.3,(a,b))
P=[w(OBS,h),np.zeros((1,h)),w(h,k),np.zeros((1,k)),
w(k,h),np.zeros((1,h)),w(h,OBS),np.zeros((1,OBS))]
m=[np.zeros_like(p) for p in P]; v=[np.zeros_like(p) for p in P]
for t in range(1, ep+1):
a1=np.tanh(X@P[0]+P[1]); cd=a1@P[2]+P[3]
a2=np.tanh(cd@P[4]+P[5]); out=a2@P[6]+P[7]; d=2*(out-X)/len(X)
g6,g7=a2.T@d, d.sum(0,keepdims=True)
d2=(d@P[6].T)*(1-a2**2); g4,g5=cd.T@d2, d2.sum(0,keepdims=True)
dc=d2@P[4].T; g2,g3=a1.T@dc, dc.sum(0,keepdims=True)
d1=(dc@P[2].T)*(1-a1**2); g0,g1=X.T@d1, d1.sum(0,keepdims=True)
for i,g in enumerate((g0,g1,g2,g3,g4,g5,g6,g7)):
m[i]=.9*m[i]+.1*g; v[i]=.999*v[i]+.001*g**2
P[i]-=lr*(m[i]/(1-.9**t))/(np.sqrt(v[i]/(1-.999**t))+1e-8)
a1=np.tanh(X@P[0]+P[1]); cd=a1@P[2]+P[3]
return cd, ((np.tanh(cd@P[4]+P[5])@P[6]+P[7]-X)**2).mean()
h2, _ = ae(X, 2)
print("what the 2-unit code contains -- correlate against the true factors:")
for i in range(2):
c = [abs(np.corrcoef(h2[:, i], factors[:, j])[0, 1]) for j in range(2)]
print(" code unit %d vs factor 0: %.4f vs factor 1: %.4f" % (i, c[0], c[1]))
print("it recovered them, but not in their original order or orientation.")
print("an autoencoder's code is only defined up to an arbitrary rotation --")
print("nothing in the loss prefers one axis over another.")
Output
The code recovered the true factors, but the ordering is arbitrary. Now compare against PCA, which solves the same reconstruction problem in closed form.
example_03.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
N, OBS = 300, 12
def standardise(a): return (a - a.mean(0)) / a.std(0)
factors = rng.normal(size=(N, 2))
mix = rng.normal(size=(2, OBS))
X = standardise(factors @ mix + rng.normal(0, 0.05, (N, OBS)))
def pca_mse(X, k):
U, s, Vt = np.linalg.svd(X - X.mean(0), full_matrices=False)
return (((U[:, :k] * s[:k]) @ Vt[:k] + X.mean(0) - X) ** 2).mean()
def ae(X, k, ep=800, lr=0.04):
r=np.random.default_rng(0); h=12; w=lambda a,b: r.normal(0,.3,(a,b))
P=[w(OBS,h),np.zeros((1,h)),w(h,k),np.zeros((1,k)),
w(k,h),np.zeros((1,h)),w(h,OBS),np.zeros((1,OBS))]
m=[np.zeros_like(p) for p in P]; v=[np.zeros_like(p) for p in P]
for t in range(1, ep+1):
a1=np.tanh(X@P[0]+P[1]); cd=a1@P[2]+P[3]
a2=np.tanh(cd@P[4]+P[5]); out=a2@P[6]+P[7]; d=2*(out-X)/len(X)
g6,g7=a2.T@d, d.sum(0,keepdims=True)
d2=(d@P[6].T)*(1-a2**2); g4,g5=cd.T@d2, d2.sum(0,keepdims=True)
dc=d2@P[4].T; g2,g3=a1.T@dc, dc.sum(0,keepdims=True)
d1=(dc@P[2].T)*(1-a1**2); g0,g1=X.T@d1, d1.sum(0,keepdims=True)
for i,g in enumerate((g0,g1,g2,g3,g4,g5,g6,g7)):
m[i]=.9*m[i]+.1*g; v[i]=.999*v[i]+.001*g**2
P[i]-=lr*(m[i]/(1-.9**t))/(np.sqrt(v[i]/(1-.999**t))+1e-8)
a1=np.tanh(X@P[0]+P[1]); cd=a1@P[2]+P[3]
return cd, ((np.tanh(cd@P[4]+P[5])@P[6]+P[7]-X)**2).mean()
print("PCA solves the same problem in closed form. on LINEAR data it is")
print("the provable optimum:")
print("%8s %16s %20s %12s" % ("dims", "PCA", "autoencoder", "winner"))
for k in (1, 2, 3):
p, a = pca_mse(X, k), ae(X, k)[1]
print("%8d %16.6f %20.6f %12s" % (k, p, a, "PCA" if p < a else "autoencoder"))
print("where the subspace is flat, PCA computes it exactly and gradient")
print("descent can only approach it. at 1 dim the autoencoder can edge")
print("ahead because a CURVED line captures more than the best straight")
print("one -- that is the whole advantage in miniature, and it is small.")
Output
On flat data PCA wins because it finds the exact solution. Give the autoencoder something curved and the picture changes.
example_04.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
N, OBS = 300, 12
f = rng.normal(size=(N, 2)); mix = rng.normal(size=(2, OBS))
Xc = np.tanh(f @ mix * 1.5) + rng.normal(0, 0.05, (N, OBS))
Xc = (Xc - Xc.mean(0)) / Xc.std(0)
def pca_mse(X, k):
U, s, Vt = np.linalg.svd(X - X.mean(0), full_matrices=False)
return (((U[:, :k] * s[:k]) @ Vt[:k] + X.mean(0) - X) ** 2).mean()
def ae(X, k, ep=800, lr=0.04):
r=np.random.default_rng(0); h=12; w=lambda a,b: r.normal(0,.3,(a,b))
P=[w(OBS,h),np.zeros((1,h)),w(h,k),np.zeros((1,k)),
w(k,h),np.zeros((1,h)),w(h,OBS),np.zeros((1,OBS))]
m=[np.zeros_like(p) for p in P]; v=[np.zeros_like(p) for p in P]
for t in range(1, ep+1):
a1=np.tanh(X@P[0]+P[1]); cd=a1@P[2]+P[3]
a2=np.tanh(cd@P[4]+P[5]); out=a2@P[6]+P[7]; d=2*(out-X)/len(X)
g6,g7=a2.T@d, d.sum(0,keepdims=True)
d2=(d@P[6].T)*(1-a2**2); g4,g5=cd.T@d2, d2.sum(0,keepdims=True)
dc=d2@P[4].T; g2,g3=a1.T@dc, dc.sum(0,keepdims=True)
d1=(dc@P[2].T)*(1-a1**2); g0,g1=X.T@d1, d1.sum(0,keepdims=True)
for i,g in enumerate((g0,g1,g2,g3,g4,g5,g6,g7)):
m[i]=.9*m[i]+.1*g; v[i]=.999*v[i]+.001*g**2
P[i]-=lr*(m[i]/(1-.9**t))/(np.sqrt(v[i]/(1-.999**t))+1e-8)
a1=np.tanh(X@P[0]+P[1]); cd=a1@P[2]+P[3]
return cd, ((np.tanh(cd@P[4]+P[5])@P[6]+P[7]-X)**2).mean()
print("the same 2 factors observed through tanh -- data on a bent surface:")
print("%8s %16s %20s %12s" % ("dims", "PCA", "autoencoder", "winner"))
wins = 0
for k in (1, 2, 3):
p, a = pca_mse(Xc, k), ae(Xc, k)[1]
wins += p > a
print("%8d %16.6f %20.6f %12s" % (k, p, a, "PCA" if p < a else "autoencoder"))
print("the autoencoder takes %d of 3 -- it bends to follow the curved" % wins)
print("surface where PCA can only pass a flat subspace through it.")
print("the summary: run PCA first. reach for an autoencoder when PCA's")
print("reconstruction error says the structure is not flat.")
print("two variants: denoising (feed corrupted input, ask for the clean")
print("one) and sparse (penalise how many code units are active instead")
print("of narrowing the layer).")
Output
Check yourself
0 of 3
Answer without scrolling back up.
Why is training a network to copy its input not a null task?
What survives the bottleneck is what the data was mostly made of. The copying is the mechanism; the constraint is the point.
What does a linear autoencoder with squared error find?
Provably the same span, though not necessarily the principal components in order. Non-linearity is what makes an autoencoder more than PCA with extra steps.
Why does a bottleneck remove noise?
Denoising autoencoders make this the training procedure - corrupt the input, ask for the clean version - which also prevents a wide code from learning the identity.
Cheat sheet
Autoencoders
Train an autoencoder to compress and reconstruct its input, examine the bottleneck, and see why a network that copies is still worth building.
DEEP LEARNING · vizlearn.in/deep_learning/autoencoders.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.