Make the latent space continuous enough to sample from, by pulling it towards a known distribution and paying for the pull.
Overview
The problem with a plain autoencoder
An autoencoder maps each input to a single point in the latent space, and nothing organises those points. Nearby codes need not decode to similar things, and the space between clusters is undefined territory.
So you can't make. Pick a random latent vector and decode it, and you almost certainly land a place the decoder was never trained, and the output is noise.
A VAE fixes this by making the latent space smooth and bounded — by build, not by hope.
Variational Autoencoders
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
A plain autoencoder maps each input to a point. A VAE maps it to a distribution.
The reparameterisation trick writes the sample as z = μ + σε, so the randomness carries no gradient.
The loss has two terms: reconstruction, and a KL that pulls every encoded distribution towards the prior.
Too much KL and the clusters merge — posterior collapse. Too little and the space has gaps, so sampling produces nothing.
Variational Autoencoders
Turning an autoencoder into a thing you can sample from. The two forces that have to be weighed to do it.
Encoding to a distribution
The encoder outputs a mean and a standard deviation rather than a point, and the code is *sampled* from that spread during training.
That single change does the work. Because the same input makes different codes on different passes, the decoder must handle a whole zone, not a point. Zones share and the space fills in.
Drag the sigma control and watch the clusters spread. At small sigma you've a plain autoencoder again, with tight clumps and gaps between them. At large sigma the zones swallow each other.
The reparameterisation trick
There's an obstacle: you can't backprop through a sampling op. Sampling isn't a differentiable function of the params that made the spread.
The trick is to move the noise out of the path:
z = mu + sigma * epsilon with epsilon ~ N(0, 1)
Now the random part, epsilon, is an input rather than an op. The gradient flows to mu and sigma most times, because with epsilon fixed the formula is an plain differentiable function of them.
That is the whole trick, it's two lines of code, and VAEs weren't trainable without it. It reappears in other places where a sample must be differentiated through — the Gumbel-softmax does the same job for discrete variables.
Two terms in tension
loss = reconstruction error + KL( encoder distribution || prior )
The rebuild term wants tight, well-separated codes: the easiest thing to decode accurately.
The KL term wants each coded spread to look like the prior — the dashed circle. It's what makes the space samplable, because if each code looks like a draw from the prior then a draw from the prior looks like a code.
They pull against each other, and the readout names which is winning.
KL winning is *posterior collapse*: the encoder outputs the prior even so of input, the clusters merge, and the decoder skips the latent fully. Raise sigma or lower the gap to see it.
Rebuild winning gives tight clusters with empty space between them — great rebuild, and sampling lands in the gaps.
The usual control is beta-VAE. This Weights the KL term by name. High beta pushes towards disentangled but blurry. Low beta towards sharp but not samplable. Many setups also *anneal* beta from zero, letting rebuild establish itself before the KL pressure arrives.
Jensen, and where the loss comes from
That loss isn't any. The quantity you really want to raise is log p(x), which holds an unsolvable integral over the latent.
Jensen's inequality converts it into a lower bound — the ELBO — which breaks down into just the two terms above. Raising the bound can't fall the true likelihood, and the gap between them is a KL divergence.
Against GANs
VAEs give a principled goal, a real latent space, stable training and a likelihood bound. Their samples are blurry, because the pixel-wise rebuild term rewards hedging: when uncertain, the mean of the possibilities scores better than any single one.
GANs give sharp samples and are far harder to train. Diffusion models have largely displaced both for image generation, and the VAE lives inside them. Latent diffusion runs the diffusion process in a VAE's latent space rather than in pixels.
The KL term, computed
A VAE's loss is rebuild error plus a KL divergence pulling each coded spread toward the unit-Gaussian prior, and for Gaussians that KL has a closed form worth seeing:
The term is zero just when the code looks like a draw from the prior, and it punishes both a mean pushed away from the origin and a variance merged toward a point — the tight, confident code a plain autoencoder loves. That is the tension: rebuild wants tight, well-separated codes it can decode just, while the KL wants each code to look prior-like so that a draw from the prior looks like a code and sampling works. When the KL wins you get *posterior collapse* — the encoder outputs the prior even so of input and the decoder skips the latent — and when rebuild wins you get sharp clusters with unsamplable gaps between them. beta-VAE weights the KL to sit the balance where you want it, and the loss itself isn't any: it's the ELBO, the Jensen's-inequality lower bound on the unsolvable log p(x).
The reparameterisation trick is what makes all of this trainable, and it's worth stating on its own: you can't backprop through a sampling step, so the code is written as z = mu + sigma · epsilon with epsilon drawn from a fixed standard normal. The noise is now an input rather than an op, so with epsilon held fixed the formula is an plain differentiable function of mu and sigma and the gradient flows to the encoder most times. It's two lines of code and VAEs didn't train without it, and the same device reappears wherever a sample must be differentiated through. The Gumbel-softmax does the the same job for discrete latents. The blur in VAE samples, by contrast, is structural: the pixel-wise rebuild term rewards hedging, so an uncertain decoder means the possibilities rather than committing to one.
Where it goes wrong with Variational Autoencoders
Posterior collapse, above all with a powerful decoder that can do well without the latent. Anneal the KL, or weaken the decoder.
Expecting sharp samples. The blur is structural.
Losing the reparameterisation trick. Without it there's no grad.
Reading latent dims as real. A plain VAE isn't disentangled; beta-VAE pushes it and doesn't ensure it.
Questions people ask about Variational Autoencoders
How is a VAE different from an autoencoder? The encoder outputs a distribution (mean and variance) rather than a single point, and a KL term keeps those distributions close to a standard normal. This makes the latent space smooth enough to sample from.
What is the reparameterisation trick? Write the sample as mean plus standard deviation times random noise. The randomness moves into an input, so gradients can flow through the mean and variance.
What are the two terms in the loss? A reconstruction term that rewards accurate output, and a KL term that regularises the latent distribution. Their balance sets the trade-off between sharpness and a well-organised latent space.
Why are VAE samples blurry? The pixel-wise reconstruction loss averages over plausible outputs. A sharper decoder loss, or a larger model, helps.
What is posterior collapse? The decoder ignores the latent code and the KL goes to zero. Fixes include KL annealing, free bits and a weaker decoder.
What is a beta-VAE? A VAE with the KL term multiplied by a factor beta. A larger beta pushes towards disentangled factors, at the cost of reconstruction quality.
The two terms in the loss, and what each one buys
A VAE is an autoencoder whose code is a spread rather than a point. That one change adds a second loss term. The balance between the two is the whole action.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
print("a plain autoencoder maps x to a POINT. a VAE maps x to a mean and a")
print("variance, then samples from it:")
mu, logvar = np.array([0.6, -1.2]), np.array([-2.0, -0.5])
sigma = np.exp(0.5 * logvar)
print(" encoder output: mu = %s, log(var) = %s" % (mu, logvar))
print(" so sigma = %s" % np.round(sigma, 4))
print()
print("the sampling has to be differentiable, which it is not if you call a")
print("random function on mu and sigma. the reparameterisation trick moves")
print("the randomness outside the path the gradient travels:")
for i in range(4):
eps = rng.normal(size=2)
print(" eps ~ N(0,1) = %s -> z = mu + sigma*eps = %s"
% (np.round(eps, 3), np.round(mu + sigma * eps, 4)))
print(" z depends on mu and sigma by plain arithmetic, so d z / d mu = 1")
print(" and d z / d sigma = eps. the gradient flows; eps is just a constant")
print(" that happens to be random.")
print()
def kl(mu, logvar):
return 0.5 * (np.exp(logvar) + mu ** 2 - 1 - logvar)
print("the second loss term is a KL divergence pulling that distribution")
print("toward a standard normal. it is zero only when mu=0 and var=1:")
print("%10s %10s %14s" % ("mu", "sigma", "KL"))
for m, s_ in ((0.0, 1.0), (0.0, 0.5), (0.0, 2.0), (2.0, 1.0), (5.0, 0.1)):
print("%10.1f %10.1f %14.6f" % (m, s_, kl(np.array([m]), np.array([2 * np.log(s_)]))[0]))
print()
print("read the last row: a code that is confidently far from the origin is")
print("expensive. that is the term stopping the encoder from doing what a")
print("plain autoencoder does -- scattering each input to its own private")
print("corner with near-zero variance.")
Output
The next block compares a plain autoencoder's codes against a VAE's to show why only one of them can generate.
example_02.pyNumPy
import numpy as np
print("build two 1-D codebooks for 5 inputs:")
plain = np.array([-9.4, -4.8, 3.7, 7.2, 11.6])
vae = np.array([-1.3, -0.6, 0.05, 0.7, 1.4])
print(" plain autoencoder codes:", plain)
print(" VAE codes :", vae)
print()
print("now sample a new point from N(0,1) -- which is what you do to GENERATE:")
for draw in (-0.5, 0.3, 1.1):
dp = np.abs(plain - draw).min()
dv = np.abs(vae - draw).min()
print(" z = %+.1f: nearest plain code %5.2f away, nearest VAE code %5.2f away"
% (draw, dp, dv))
print()
print(" every draw lands in a region the plain decoder has never been shown")
print(" -- its codes are all several units away. it will produce noise. the")
print(" VAE's decoder has seen the whole of that region, because the KL term")
print(" packed every code into exactly the distribution you are sampling")
print(" from. that is what makes one of them a generative model.")
print()
print("that is the entire difference between an autoencoder and a GENERATIVE")
print("model: not the architecture, but whether the space between the codes")
print("decodes to anything sensible.")
Output
The last block shows the beta dial, posterior collapse, and why VAE samples look soft.
example_03.pyNumPy
import numpy as np
def kl(mu, logvar):
return 0.5 * (np.exp(logvar) + mu ** 2 - 1 - logvar)
print("the balance between the two terms is a real dial:")
print("%10s %28s %26s" % ("beta", "what wins", "what you get"))
rows = [(0.0, "reconstruction only", "a plain autoencoder"),
(0.1, "reconstruction mostly", "sharp, poorly organised space"),
(1.0, "the true ELBO", "the standard VAE"),
(5.0, "KL mostly", "well-organised, blurry"),
(50.0, "KL only", "posterior collapse")]
for b, w, g in rows:
print("%10.1f %28s %26s" % (b, w, g))
print()
print("posterior collapse is the failure worth naming. if the KL term")
print("dominates, the cheapest solution is for the encoder to ignore x")
print("entirely and output mu=0, sigma=1 for every input:")
print(" KL for that = %.6f, which is the minimum possible."
% kl(np.zeros(2), np.zeros(2)).sum())
print(" the code now carries no information, and the decoder learns to")
print(" produce the dataset average for everything. the reconstruction")
print(" loss is terrible and the KL is perfect.")
print()
print("and the reason VAE samples look soft: the reconstruction term is")
print("usually a squared error, and squared error is minimised by the MEAN")
print("of all plausible outputs. averaging several sharp possibilities gives")
print("you one blurry one. that is a property of the loss, not the model.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
What does the encoder of a VAE output?
The code is sampled from that distribution, so the same input gives different codes on different passes and the decoder must handle a neighbourhood rather than a point.
What problem does the reparameterisation trick solve?
z = mu + sigma*epsilon makes the expression an ordinary differentiable function of mu and sigma with epsilon fixed. VAEs were not trainable without it.
What is posterior collapse?
The opposite failure - reconstruction winning - gives tight clusters with empty space between them, where sampling lands in the gaps.
Cheat sheet
Variational Autoencoders
Add a KL divergence term to an autoencoder loss, sample from the latent space, and see how two competing objectives produce a generative model.
DEEP LEARNING · vizlearn.in/deep_learning/variational_autoencoders.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.