Learn what things mean without labels, by insisting that two views of the same thing land together and everything else lands apart.
Overview
The pretext
Labels cost money. Raw data is free. Self-taught learning makes a task from the data itself, one that forces the model to grasp the content.
The contrastive version is simple. Take an image, make two changed views, and train the model to place them close. Push every other image's views far away.
If the model can tell that a crop and a colour-shifted copy of the same photo go together — and tell them apart from a thousand other photos — it has had to learn what the photo is *of*.
Contrastive and Self-Supervised Learning
This module needs JavaScript: the
numbers are computed in the page rather than recorded.
Worth knowing
Two augmentations of one image are a positive pair. Everything else in the batch is a negative.
InfoNCE is a softmax over similarities: it is cross-entropy where the classes are 'which of these is the match'.
Temperature decides how much the hardest negatives dominate. It is the parameter that matters most.
The augmentations define what the model is told to ignore. Choose them badly and you teach it the wrong invariance.
Contrastive and Self-Supervised Learning
Getting a useful shape out of raw data by turning the data into its own teacher.
InfoNCE
The loss is a softmax over likenesses:
loss = -log[ exp(sim(a, p) / tau)
/ ( exp(sim(a, p) / tau) + sum over negatives exp(sim(a, n) / tau) ) ]
It is cross-entropy. The classes are "which of these items is the match" and the right answer is the positive. All you know about cross-entropy applies here.
The chart puts the anchor and its positive on the unit circle with the negatives around them, and shows the real loss. Move the jitter slider and the positive drifts from the anchor. A harder task raises the loss.
Temperature
tau is small and it matters more than anything else in the loss.
Low temperature sharpens the softmax. The sum is ruled by the single closest negative. The gradient focuses on that hardest case, which learns fine gaps quickly but is unstable.
High temperature flattens the scores until all negatives count the same, and the loss stops being useful.
Drag it across its range. The loss moves by a factor of ten with no change to the data at all.
Negatives, and doing without them
More negatives usually means a better model. The task is harder, so what the model learns is more useful. The readout shows the loss rising with the count for that reason.
That made a real problem: negatives come from the batch, and batches have limits. SimCLR used very large batches. MoCo kept a queue of past outputs with a slowly-updated encoder.
Then a second family showed negatives were not needed at all. BYOL and SimSiam use only positive pairs. They avoid collapse — where the encoder maps all inputs to one point — with a trick: a predictor head on one branch and a stop-gradient on the other. Barlow Twins and VICReg add a term that keeps the output dims spread apart.
That negatives turned out to be optional was a real surprise. Why it works is still debated.
Augmentation is the design
The boosts decide what the model learns to ignore, and so what it learns.
Colour jitter teaches colour steadiness. That is great for finding objects and wrong when colour is the label. Random cropping teaches that a part implies the whole. That is most of why contrastive learning works on images. It is also why it can match textures instead of objects.
SimCLR's tests found the boost choice mattered more than the model itself. That was a surprise and it held up.
Where it is used
Pretraining on untagged data, then fine-tuning on a small labelled set. This is the main use, and it's why the field cared.
CLIP contrasts images against their captions rather than against other views, which is what made a model that can be prompted in language.
Sentence embeddings. SimCSE contrasts a sentence against itself under dropout.
Recommendation and lookup, where "these two things go together" is the native form of the data.
InfoNCE by hand, and why temperature dominates
The contrastive loss is plain cross-entropy over "which item is the match." Try it by hand. Anchor score to the positive is 0.9. Scores to four negatives are 0.85 (one hard), 0.2, 0.1, 0.3:
temperature tau
loss
P(picks the positive)
0.05
0.31
0.73
0.10
0.48
0.62
0.50
0.98
0.38
Same data, same scores. Only tau changed. The loss moves by a factor of three. Low temperature sharpens the softmax so the hard negative at 0.85 eats most of the gradient. High temperature flattens the scores until all negatives count the same.
That is why you tune temperature before anything else. SimCLR's tests found it and the boost choice mattered more than the encoder. More negatives raise the loss for the right reason: a harder task helps more. That pushed the field to large batches and queues, before BYOL and SimSiam showed negatives could be dropped.
The boosts are not a prep step. They *are* the design. They set what the model learns to treat as noise. Colour jitter teaches colour steadiness, which is right for finding objects and wrong when colour is the label. Random cropping teaches that a part implies the whole.
The other real risk is false negatives. Two photos of the same class get pushed apart by a loss that has no labels. That cost is real, and guided contrastive learning fixes it when labels are on hand. CLIP is the variant that matches images to their captions instead of to other views. That is what made a model you can prompt with words.
Where it goes wrong with Contrastive and Self-Supervised Learning
Boosts that erase the label. Colour jitter on a task about colour.
Default temperature. Tune it. It does more than the model does.
Too few negatives without a method built for that (BYOL, SimSiam).
False negatives. Two photos of the same class are pushed apart. The loss has no labels and cannot know they belong together. Guided contrastive learning fixes this when labels are on hand.
Questions people ask about Contrastive and Self-Supervised Learning
What is contrastive learning? A way to learn representations without labels by pulling together views of the same example and pushing apart different ones.
What is the InfoNCE loss? A softmax cross-entropy over similarities. The positive pair should score higher than all negatives in the batch.
What does the temperature do? It scales the similarities before the softmax. A low temperature sharpens the loss and focuses it on the hardest negatives, a high one treats all negatives alike.
Why does batch size matter? Other examples in the batch serve as negatives, so a larger batch gives more of them. Methods like MoCo use a queue to get many negatives with small batches.
Can it work without negatives? Yes. BYOL and SimSiam use a predictor and stop-gradient to avoid collapse, with no negative pairs.
Why is augmentation so important? Augmentations define what counts as the same thing. Crops, colour jitter and blur tell the model which changes to ignore, and so shape what it learns.
Learn what is similar without a single label
Contrastive learning pulls two views of the same thing together and pushes all else apart. Here is the InfoNCE loss found by hand, then trained -- and the class structure that appears without any class labels.
example_01.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
print("the loss, on four items with two views each. z is a batch of")
print("normalised embeddings; item i's positive is item i's other view.")
def normalise(z):
return z / np.linalg.norm(z, axis=1, keepdims=True)
a = normalise(rng.normal(size=(4, 8)))
b = normalise(a + rng.normal(0, 0.25, (4, 8)))
sim = a @ b.T
print("similarity matrix (view 1 rows against view 2 columns):")
print("%8s %8s %8s %8s %8s" % ("", "b0", "b1", "b2", "b3"))
for i in range(4):
print("%8s %8.4f %8.4f %8.4f %8.4f" % ("a%d" % i, *sim[i]))
print(" the DIAGONAL should be large -- those are matching pairs.")
print(" every off-diagonal entry is a negative.")
TAU = 0.1
def info_nce(sim, tau=TAU):
logits = sim / tau
logits = logits - logits.max(axis=1, keepdims=True)
p = np.exp(logits) / np.exp(logits).sum(axis=1, keepdims=True)
return -np.log(np.diag(p)).mean(), p
loss, p = info_nce(sim)
print("divide by temperature, then softmax each ROW:")
print("%8s %8s %8s %8s %8s" % ("", "b0", "b1", "b2", "b3"))
for i in range(4):
print("%8s %8.4f %8.4f %8.4f %8.4f" % ("a%d" % i, *p[i]))
print(" that is cross-entropy where the 'correct class' for row i is")
print(" column i. InfoNCE is a classification loss over the batch.")
print(" loss = %.4f" % loss)
Output
Temperature and batch size control how contrastive learning uses its negatives.
example_02.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
def normalise(z):
return z / np.linalg.norm(z, axis=1, keepdims=True)
a = normalise(rng.normal(size=(4, 8)))
b = normalise(a + rng.normal(0, 0.25, (4, 8)))
sim = a @ b.T
def info_nce(sim, tau):
logits = sim / tau
logits = logits - logits.max(axis=1, keepdims=True)
p = np.exp(logits) / np.exp(logits).sum(axis=1, keepdims=True)
return -np.log(np.diag(p)).mean(), p
print("TEMPERATURE controls how hard the negatives are pushed:")
print("%10s %14s %26s" % ("tau", "loss", "weight on hardest negative"))
for tau in (1.0, 0.5, 0.1, 0.05, 0.01):
l, pp = info_nce(sim, tau)
off = pp.copy(); np.fill_diagonal(off, 0)
print("%10.2f %14.4f %26.4f" % (tau, l, off.max()))
print(" small tau sharpens the softmax, so almost all gradient goes to")
print(" the single closest negative. large tau spreads it evenly.")
print(" 0.05 to 0.1 is the usual range -- not a throwaway constant.")
print("BATCH SIZE matters for the same reason. every other item in the")
print("batch is a negative, so the batch IS your supply of them:")
for n in (8, 64, 256, 4096):
print(" batch %5d -> %5d negatives per positive" % (n, n - 1))
print(" this is why contrastive methods use batches of thousands,")
print(" and why MoCo keeps a queue of past embeddings instead.")
Output
Now train a contrastive model on synthetic data with three latent clusters and check whether the learned embedding ignores the nuisance dimensions.
example_03.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
K,DIM,OUT,TAU,N = 3,16,8,0.1,240
centres = rng.normal(size=(K,DIM))*1.2
labels = rng.integers(0,K,N); CD=8
content = np.zeros((N,DIM))
content[:,:CD] = centres[labels][:,:CD] + rng.normal(0,.4,(N,CD))
def view_of(rows):
v=content[rows].copy(); v[:,CD:]=rng.normal(0,2.5,(len(rows),DIM-CD)); return v
P=[rng.normal(0,.4,(DIM,32)),np.zeros(32),rng.normal(0,.4,(32,OUT)),np.zeros(OUT)]
m=[np.zeros_like(x) for x in P]; v=[np.zeros_like(x) for x in P]
def encode(x):
h=np.tanh(x@P[0]+P[1]); z=h@P[2]+P[3]; n=np.linalg.norm(z,axis=1,keepdims=True)+1e-9
return z/n, h, z, n
print("training: 3 latent clusters, two noisy views, NO class labels:")
B=64
for t in range(1,2001):
ix=rng.integers(0,N,B); v1,v2=view_of(ix),view_of(ix)
z1,h1,_,n1=encode(v1); z2,h2,_,n2=encode(v2)
lg=z1@z2.T/TAU; lg-=lg.max(1,keepdims=True)
pr=np.exp(lg); pr/=pr.sum(1,keepdims=True)
if t%400==0 or t==1:
print(" step %4d: loss %.4f (chance %.4f)" % (t,-np.log(np.diag(pr)+1e-12).mean(),np.log(B)))
d=pr.copy(); d[np.arange(B),np.arange(B)]-=1; d/=B*TAU
for gz,z,h,n,x in ((d@z2,z1,h1,n1,v1),(d.T@z1,z2,h2,n2,v2)):
gr=(gz-(gz*z).sum(1,keepdims=True)*z)/n
g2,g3=h.T@gr,gr.sum(0); dh=(gr@P[2].T)*(1-h**2); g0,g1=x.T@dh,dh.sum(0)
for i,g in enumerate((g0,g1,g2,g3)):
m[i]=.9*m[i]+.1*g; v[i]=.999*v[i]+.001*g**2
P[i]-=.01*(m[i]/(1-.9**t))/(np.sqrt(v[i]/(1-.999**t))+1e-8)
va,vb = view_of(np.arange(N)), view_of(np.arange(N))
Za,Zb = encode(va)[0], encode(vb)[0]
ra = va/np.linalg.norm(va,axis=1,keepdims=True)
rb = vb/np.linalg.norm(vb,axis=1,keepdims=True)
print("VIEW INVARIANCE -- how similar are two views of the SAME item?")
print(" raw observation : %.4f" % np.mean(np.sum(ra*rb, axis=1)))
print(" learned embedding : %.4f" % np.mean(np.sum(Za*Zb, axis=1)))
print(" the raw vectors agree weakly because half is fresh noise.")
print(" the embedding learned to throw that half away.")
Output
The embedding ignores the nuisance half. But does it also discover the cluster structure?
example_04.pyNumPy
import numpy as np
rng = np.random.default_rng(0)
K,DIM,OUT,TAU,N = 3,16,8,0.1,240
centres = rng.normal(size=(K,DIM))*1.2
labels = rng.integers(0,K,N); CD=8
content = np.zeros((N,DIM))
content[:,:CD] = centres[labels][:,:CD] + rng.normal(0,.4,(N,CD))
def view_of(rows):
v=content[rows].copy(); v[:,CD:]=rng.normal(0,2.5,(len(rows),DIM-CD)); return v
P=[rng.normal(0,.4,(DIM,32)),np.zeros(32),rng.normal(0,.4,(32,OUT)),np.zeros(OUT)]
m=[np.zeros_like(x) for x in P]; v=[np.zeros_like(x) for x in P]
def encode(x):
h=np.tanh(x@P[0]+P[1]); z=h@P[2]+P[3]; n=np.linalg.norm(z,axis=1,keepdims=True)+1e-9
return z/n, h, z, n
B=64
for t in range(1,2001):
ix=rng.integers(0,N,B); v1,v2=view_of(ix),view_of(ix)
z1,h1,_,n1=encode(v1); z2,h2,_,n2=encode(v2)
lg=z1@z2.T/TAU; lg-=lg.max(1,keepdims=True)
pr=np.exp(lg); pr/=pr.sum(1,keepdims=True)
d=pr.copy(); d[np.arange(B),np.arange(B)]-=1; d/=B*TAU
for gz,z,h,n,x in ((d@z2,z1,h1,n1,v1),(d.T@z1,z2,h2,n2,v2)):
gr=(gz-(gz*z).sum(1,keepdims=True)*z)/n
g2,g3=h.T@gr,gr.sum(0); dh=(gr@P[2].T)*(1-h**2); g0,g1=x.T@dh,dh.sum(0)
for i,g in enumerate((g0,g1,g2,g3)):
m[i]=.9*m[i]+.1*g; v[i]=.999*v[i]+.001*g**2
P[i]-=.01*(m[i]/(1-.9**t))/(np.sqrt(v[i]/(1-.999**t))+1e-8)
va = view_of(np.arange(N)); Za = encode(va)[0]
ra = va/np.linalg.norm(va,axis=1,keepdims=True)
def sep(M):
s=np.mean([M[i]@M[j] for i in range(N) for j in range(N) if i!=j and labels[i]==labels[j]])
return s, np.mean([M[i]@M[j] for i in range(N) for j in range(N) if labels[i]!=labels[j]])
rs, rd = sep(ra); es, ed = sep(Za)
print("CLUSTER STRUCTURE:")
print("%22s %16s %16s %12s" % ("", "same cluster", "different", "separation"))
print("%22s %16.4f %16.4f %12.4f" % ("raw view", rs, rd, rs-rd))
print("%22s %16.4f %16.4f %12.4f" % ("learned embedding", es, ed, es-ed))
print("the embedding did NOT separate clusters better. InfoNCE is")
print("INSTANCE discrimination -- same-class items in the batch are")
print("FALSE NEGATIVES, actively pushed apart.")
Output
The gap did not grow. InfoNCE sorts by instance, not by class — it was never told which items share a label. Same-class items in the batch are pushed apart as false negatives.
That is why the field added guided contrastive loss (same-label items count as positives), careful boost design (the boosts *define* what counts as same), and linear probes for testing instead of direct clusters. The whole point is that a task built from raw data makes a form that a small labelled set can then tune cheaply.
The other failure mode is collapse.
example_05.pyNumPy
import numpy as np
B = 64
print("the failure mode is COLLAPSE -- mapping every input to the same")
print("vector makes all positives perfectly similar:")
print(" if every z were identical, diagonal similarity would be 1.0")
print(" and the loss would be log(batch) = %.4f" % np.log(B))
print(" the negatives prevent it: they are also similarity 1.0, so")
print(" the softmax cannot favour the diagonal. remove negatives and")
print(" collapse is the global optimum -- which is why BYOL and SimSiam")
print(" need a stop-gradient or predictor head instead.")
Output
Check yourself
0 of 3
Answer without scrolling back up.
What is a positive pair in contrastive learning?
No labels are involved, which is the point. Two images of the same class are treated as negatives - a real cost that supervised contrastive learning fixes when labels exist.
What does a low temperature do to InfoNCE?
It learns fine distinctions quickly and is unstable. High temperature flattens everything until the loss stops discriminating at all.
How do BYOL and SimSiam avoid collapse without negatives?
The large-batch and queue approaches are SimCLR and MoCo, which do use negatives. That negatives turned out to be optional was a genuine surprise.
Cheat sheet
Contrastive and Self-Supervised Learning
Train a model to pull matching pairs together and push non-matches apart without labels, and see a useful representation emerge from raw data.
DEEP LEARNING · vizlearn.in/deep_learning/contrastive_learning.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.