Quick Context
Reproducibility means getting the exact same results when you run the same code with the same data. In deep learning, multiple sources of randomness — weight initialization, data shuffling, dropout masks, GPU floating-point non-determinism — make this surprisingly hard. Ensuring reproducibility is critical for debugging, research, and production deployment.
1) Sources of Randomness
- Weight initialization — random initial values differ between runs.
- Data shuffling — different batch orderings change gradient updates.
- Dropout — different neurons are randomly dropped each forward pass.
- Data augmentation — random crops, flips, and color jitter produce different training samples.
- GPU non-determinism — parallel floating-point operations don't guarantee operation order, causing tiny numerical differences that compound.
- Library versions — different versions of PyTorch, CUDA, or cuDNN may use different algorithms.
2) How to Set Seeds Properly
import random, numpy as np, torch
SEED = 42
random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)
torch.cuda.manual_seed_all(SEED)
# For full GPU determinism (may slow training):
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
You must seed all random number generators: Python's built-in, NumPy, and PyTorch (both CPU and CUDA).
3) Guided Experiments
- Train twice without setting a seed — observe that final weights and loss differ.
- Set the same seed and train twice — results should be identical.
- Change only the seed value and train — results differ, proving the seed controls randomness.
- Train with the same seed but different hyperparameters — observe that the seed doesn't "fix" the model to be good; it just makes randomness repeatable.
4) Beyond Seeds: Full Reproducibility Checklist
- Pin library versions in requirements.txt or environment.yml.
- Use deterministic algorithms (PyTorch:
torch.use_deterministic_algorithms(True)).
- Fix the number of DataLoader workers (or use workers=0 for full determinism).
- Log the seed and all hyperparameters with every experiment.
- Use version control for code and DVC for data/model artifacts.
- Use Docker containers for environment reproducibility across machines.
5) Common Mistakes
- Setting only one seed (e.g., just NumPy) and forgetting the others.
- Expecting GPU reproducibility without setting deterministic flags — GPU parallelism introduces non-determinism by default.
- Relying on a "lucky seed" — if your result only works with one specific seed, there's a bigger problem with your model or data.
6) Key Takeaways
- Deep learning has many sources of randomness; seed ALL of them.
- GPU determinism requires explicit flags and may slow training.
- True reproducibility = seeds + pinned versions + documented environment.
- A good model should work across different seeds; reproducibility is for debugging, not cherry-picking.