vizlearn
Modules / Deep Learning / Transfer Learning

What is Transfer Learning in CNN?

Computer VisionBy Updated

Select a pre-trained model, choose a strategy, and selectively unfreeze layers to see how fine-tuning impacts the learning curve and final accuracy.

Overview

What is Transfer Learning?

Training a Deep Convolutional Neural Network from scratch requires millions of images (like the ImageNet dataset), massive amounts of computing power (GPUs), and weeks of training time. Furthermore, early layers in a CNN generally learn the exact same things regardless of the dataset: generic edges, colors, and basic textures.

Transfer Learning is the practice of taking a pre-trained model (whose weights have already been optimized on a massive dataset) and repurposing it for a new, related task. You are transferring the "knowledge" of visual features to a new domain.

Model Architecture

Click Conv layers to toggle Freezing

Understanding Transfer Learning

Why reinvent the wheel? Learn how to reuse powerful models trained by giant tech companies for your own custom tasks.

How it Works: Modifying the Architecture

A standard pre-trained model (like ResNet-50 or VGG-16) is split into two conceptual parts:

  • The Convolutional Base: Extracts features. We keep this part and its pre-trained weights.
  • The Classification Head (Dense Layers): Outputs predictions for the original 1000 classes. We remove this part, because we want to predict our own classes (e.g., 2 classes for Medical X-Rays).

After removing the original head, we attach a new, randomly initialized Dense layer matching our target class count. Now, we have to decide how to train this new hybrid model.

The Three Strategies

  1. Train from Scratch:

    We ignore pre-trained weights and initialize the entire network randomly. The network has no prior knowledge. The accuracy climbs very slowly, and on a small dataset, heavy models like VGG-16 will likely overfit and fail to reach high performance.

  2. Feature Extraction (Freeze the Base):

    We load the pre-trained weights into the Conv Base and freeze them (set them to untrainable). During training, only the weights of our newly added Dense Head are updated. The frozen base acts as a powerful feature extractor. Training is extremely fast, and accuracy jumps up quickly because the network already knows how to "see".

  3. Fine-Tuning (Selective Unfreezing):

    First, we perform Feature Extraction. Then, we unfreeze specific layers of the Convolutional Base (usually the top layers) and train them alongside the Dense Head using a very small learning rate. (In the app, select Fine-Tune and click layers to unlock them). We keep early layers frozen (generic edges), but let later layers adapt their specific shape-detectors to our custom images. This pushes the accuracy to its absolute maximum limit.

Catastrophic Forgetting

In Fine-Tuning mode, try unfreezing all the Convolutional layers. You will notice the accuracy curve might drop or cap lower than expected. If you unfreeze the entire network and train it on a small dataset, the massive gradients from the randomly initialized Dense Head propagate backward and wreck the delicate, pre-trained weights of the entire base. This is called Catastrophic Forgetting.

Reusing what another model already learned

Training a vision model from scratch needs a great deal of data. ImageNet has 1.2 million labelled images; most projects have a few hundred.

Transfer learning closes that gap. Take a network already trained on a large dataset, keep the features it learned, and adapt only the parts that are specific to your task.

The reason it works is what the layers contain. Early layers learn edges, colours and textures — visual primitives that are useful for essentially any image task, whether the subject is animals, X-rays or circuit boards. Only the final layers are specific to the original classes.

So the recipe is: keep the general part, replace the specific part.

Layer depthWhat it learnedReusable?
FirstEdges, colour transitionsAlmost always
Early middleTextures, corners, simple shapesUsually
Late middleObject partsOften, with fine-tuning
FinalThe original 1,000 classesNo — replace it

Two strategies, and when to use each

Feature extraction. Freeze the entire pretrained network, replace the classifier head, and train only the head. Fast, needs little data, and very hard to overfit.

Fine-tuning. Train the head first, then unfreeze some or all of the backbone and continue with a much lower learning rate. Slower, needs more data, and reaches higher accuracy when your images differ from the original dataset.

The choice depends on two things — how much data you have, and how similar it is to the pretraining data:

 Similar to ImageNetVery different (X-rays, satellite)
Little dataFreeze everything, train the headFreeze early layers, train the later ones
Plenty of dataFine-tune the last few blocksFine-tune everything, or train from scratch

The learning rate is what makes fine-tuning work or fail. Use something like 10–100× lower than you would for training from scratch — the pretrained weights are already good, and a large step destroys them in the first few batches. Discriminative learning rates, where earlier layers get smaller rates than later ones, are a standard refinement.

Doing it in code

import torch
import torch.nn as nn
from torchvision import models

model = models.resnet50(weights="IMAGENET1K_V2")

for p in model.parameters():          # freeze the backbone
    p.requires_grad = False

model.fc = nn.Linear(model.fc.in_features, num_classes)   # new head, trainable

opt = torch.optim.AdamW(model.fc.parameters(), lr=1e-3)
# ... train the head for a few epochs ...

for p in model.layer4.parameters():   # then unfreeze the last block
    p.requires_grad = True

opt = torch.optim.AdamW([
    {"params": model.layer4.parameters(), "lr": 1e-5},
    {"params": model.fc.parameters(),     "lr": 1e-4},
])

Two details that are easy to get wrong. Replacing model.fc creates a new layer with requires_grad=True by default, which is what you want — but check the trainable parameter count rather than assuming. And keep batch normalisation layers in evaluation mode while the backbone is frozen, or their running statistics will drift on your small dataset and quietly degrade the features.

The short of it

  • Transfer Learning reuses Convolutional layers from models trained on huge datasets.
  • Freezing a layer means its weights are locked and not updated during backpropagation.
  • Optimal Fine-Tuning usually involves unfreezing only the top 1 or 2 Convolutional blocks.
  • Different architectures (VGG vs ResNet) have vastly different parameter counts and training speeds.

Preprocessing must match the original

A pretrained network expects its input to look statistically like what it was trained on. Three things have to match:

Normalisation. ImageNet models expect per-channel standardisation with mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225]. Feeding 0–1 images without this shifts every activation and costs several points of accuracy for no obvious reason.

Input size. Usually 224×224. Modern architectures tolerate other sizes thanks to global average pooling, but the features were learned at a particular scale and work best near it.

Channel order. Torchvision expects RGB; OpenCV loads BGR. This mismatch produces a model that works but performs oddly, and it is a genuinely common bug.

The frameworks now ship the correct transforms alongside the weights — weights.transforms() in torchvision — and using that is safer than copying constants from a blog post.

Which backbone to start from

BackboneCharacter
ResNet-50The reliable default; well understood, widely supported
EfficientNetBetter accuracy per FLOP; good on constrained hardware
MobileNetV3Built for phones and edge devices
ConvNeXtModern convolutional design, transformer-inspired
ViT / SwinStrong with large data and large pretraining
CLIP encodersExcellent general-purpose features, and zero-shot capable

For a first attempt, ResNet-50 fine-tuned from ImageNet weights is hard to beat as a baseline. CLIP-pretrained backbones have become a strong alternative because their features transfer unusually well to domains far from ImageNet.

Self-supervised pretraining (SimCLR, DINO, MAE) matters when you have a lot of unlabelled domain images: pretrain on those without labels, then fine-tune on your small labelled set. In specialised domains this often beats ImageNet initialisation outright.

When transfer learning does not help

Very different data. Medical scans, satellite imagery, microscopy and audio spectrograms differ enough from natural photographs that the benefit shrinks — though it is usually still positive, and still worth trying first.

Very small images. A network built for 224×224 downsamples 32×32 CIFAR images to nothing. Use an architecture designed for the resolution.

Plenty of data. With millions of domain images, training from scratch can exceed a fine-tuned model. Below that, it rarely does.

Catastrophic forgetting. Fine-tuning with too high a learning rate wipes out the pretrained features in a few hundred steps. If accuracy drops sharply at the start of fine-tuning, the learning rate is the first thing to check.

Reusing features someone else paid for

Transfer learning replaces the last layer of a trained network and reuses everything before it. This measures why that works, when it stops working, and the two settings -- which layers to freeze and what learning rate to use -- that decide whether you get a good model or a broken one.

example_01.pyNumPy
Output

                

                

                    
Questions people ask

                    
How many images do I need? With a frozen backbone, a few dozen per class can work. For fine-tuning, a few hundred per class is a reasonable target.
Should I freeze batch normalisation? Yes when the backbone is frozen — keep those layers in eval mode. When fine-tuning the whole network on a reasonable batch size, let them update.
Which layers should I unfreeze first? The last block. Work backwards if accuracy is still improving and you have the data to support it.
Can I transfer between very different tasks? Yes — ImageNet features are a good starting point for detection, segmentation and retrieval, not only classification.
Is a bigger backbone always better? No. With a small dataset a large model overfits faster, and inference cost may matter more than the last point of accuracy.
Why is my fine-tuned model worse than the frozen one? Almost always the learning rate is too high, or the dataset is too small to support updating that many parameters.

                

                

                    
Recap in one screen

                    
  • Reuse a network trained on a large dataset: keep the general features, replace the task-specific head.
  • Freeze everything for small or similar datasets; fine-tune the later blocks when data allows.
  • Use a much lower learning rate for pretrained weights than you would from scratch.
  • Match the original preprocessing exactly — normalisation, size and channel order.
  • ImageNet weights transfer surprisingly far, and self-supervised pretraining on your own unlabelled images goes further still.

                

            


        

    



        
    



Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
  1. What does this module say about “What is Transfer Learning”?
  2. What does this module say about “How it Works: Modifying the Architecture”?
  3. What does this module say about “The Three Strategies”?





 Cheat sheet
Transfer Learning with CNN
Training a Deep Convolutional Neural Network from scratch requires millions of images (like the ImageNet dataset), massive amounts of computing power (GPUs), and weeks of training time. Furthermore, early layers in a CNN generally learn the exact same things regardless of the dataset: generic edges, colors, and basic textures.
COMPUTER VISION · vizlearn.in/computer_vision/transfer_learning_with_cnn.html
More in Computer Vision
Computer VisionVGG-16Computer VisionInceptionNetComputer VisionAnchor BoxesComputer VisionIoU and Non-Max Suppression
Further reading
About the author
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.
About VizLearnGitHubLinkedInContact