True reproducibility means fixing the Random Seed controls both the initial network weights and the order in which data is shuffled during training. Change the seed to see the data sequence entirely rearrange, then switch back to Seed 42 to verify identical structural recovery!
Determinism Checks
Active Seed Environment
SEED: 42
Determinized Data Pipeline
Sample #0
The Seed controls the random shuffle order.
Weight Hash
--
Sum of pseudo-random weights.
Activation Hash
--
Dependent on specific data sample.
PRO TIP: Memorize the exact data order (D1..D12) for Seed 42. Change the seed to scramble the order, then change it back to verify identical shuffling!
Drag to Pan | Scroll to Zoom
Overview
Where the randomness comes from
Neural network training is randomised in at least four independent places. And each one of them changes the final weights:
Weight initialisation. Each weight starts as a random draw. Other draws land the optimiser in other basins of the loss surface.
Data shuffling. Mini-batches are formed from a shuffled dataset. So a other order means other gradients at each step.
Dropout. Which units are zeroed is resampled on each forward pass in training.
Augmentation. Random crops, flips and noise mean the net literally never sees the same input twice.
None of these is a defect — each is deliberately there to improve generalisation. But they mean “the net” is a sample from a distribution of nets, not a fixed object.
This page shows how to seed each source of randomness in PyTorch, NumPy. And Python's random module so that a training run is deterministic, what the CUBLAS_WORKSPACE_CONFIG environment var does and why deterministic mode has a real performance cost. And how to measure the variance across seeds so you know whether a 0.3% improvement is a real gain or a lucky draw. The code blocks below train the same small network five times and print the spread. It on most hardware is more than people expect.
True reproducibility means fixing the Random Seed controls both the initial network weights and the order in which data is shuffled during training.
Train the same model twice on the same data and you get two different results. Controlling the randomness is what turns a demo into an experiment you can trust.
Where the randomness comes from
Neural network training is randomised in at least four independent places. And each one of them changes the final weights:
Weight initialisation. Each weight starts as a random draw. Other draws land the optimiser in other basins of the loss surface.
Data shuffling. Mini-batches are formed from a shuffled dataset. So a other order means other gradients at each step.
Dropout. Which units are zeroed is resampled on each forward pass in training.
Augmentation. Random crops, flips and noise mean the net literally never sees the same input twice.
None of these is a defect — each is deliberately there to improve generalisation. But they mean “the net” is a sample from a distribution of nets, not a fixed object.
This page shows how to seed each source of randomness in PyTorch, NumPy. And Python's random module so that a training run is deterministic, what the CUBLAS_WORKSPACE_CONFIG environment var does and why deterministic mode has a real performance cost. And how to measure the variance across seeds so you know whether a 0.3% improvement is a real gain or a lucky draw. The code blocks below train the same small network five times and print the spread. It on most hardware is more than people expect.
Seeding, and what a seed really fixes
These are all pseudo-random: a deterministic sequence made from a starting value called the seed. Fix the seed and the sequence is identical each run.
random.seed(42) # Python’s own generator
np.random.seed(42) # NumPy
torch.manual_seed(42) # PyTorch, CPU and GPU
The catch is that these are split generators. Seeding NumPy does nothing for PyTorch’s dropout. And seeding PyTorch does nothing for a shuffle done with Python’s random. Each library that draws a random count needs its own seed. And missing one is the usual reason a “seeded” run is still not reproducible.
Why the same code gives other counts
Run the same training script twice and the results differ. Five sources of randomness are involved. And each has to be pinned apart.
Source
What varies
Weight initialisation
The starting point
Data shuffling
The order and composition of batches
Dropout
Which units are zeroed each step
Augmentation
The random transformations applied
GPU kernels
Floating-point summation order
The first four are ordinary pseudo-random count generators and can be seeded. The fifth is other in kind: many GPU steps sum vals in a non-deterministic order. And floating-point addition is not associative. So (a + b) + c and a + (b + c) can differ in the last bits. Those gaps amplify over thousands of steps.
Seeding gets you most of the way. Full determinism costs a little speed.
Pinning it down
import os, random, numpy as np, torch
def set_seed(seed=42):
os.environ["PYTHONHASHSEED"] = str(seed)
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
set_seed(42)
# for full determinism, at some cost in speed
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.use_deterministic_algorithms(True)
Two extra details that are easy to miss.
Data loader workers need their own seeding. Each worker process inherits a other base seed. And without worker_init_fn a few workers can generate identical augmentation sequences — or other ones between runs.
cudnn.benchmark = True is non-deterministic by design. It profiles a few convolution algorithms and picks the fastest. It can differ between runs. Leaving it on is often worth 10–20% speed. Turning it off is needed for bit-exact reproducibility.
Reproducibility is more than the seed
A fixed seed makes one script repeatable on one machine. Reproducing a result months later, or on someone else's machine, needs more.
Pin the environment. Library versions change numerical behaviour and at times defaults. Record exact versions — requirements.txt with pinned versions, a lock file. Or a container pic.
Version the data. "The dataset" is a moving target if it is a folder someone adds files to. Record a hash, a snapshot, or use a data-versioning tool.
Log the setup. Each hyperparameter, the seed, the git commit. And the command line. A result you cannot map back to its settings is not a result.
Save checkpoints with their config. A weights file alone does not tell you what preprocessing it expects.
Tools such as MLflow, Weights & Biases and DVC exist to automate this. And a disciplined text log plus git tags covers most of it.
Interactive Exploration Guide
Fix the seed and repeat. Enter a seed value and press Apply Seed & Init, note the initial weights, then press it again with the same seed. Identical vals — the initialisation is fully found by that count.
Change one digit. Change the seed and re-initialise. Each weight is other. The seed is not a setting with a meaningful scale. It just selects an arbitrary point in the sequence.
Watch the sequence advance. Press Next Sample a few times. Each draw differs, but the whole run of draws replays identically after re-applying the same seed — that is what pseudo-random means.
Stream without reseeding. Press Auto Stream and let it run, then re-apply the seed. The stream restarts from the beginning of the same sequence, not from where it stopped.
Why the GPU can still be non-deterministic
Seeding each generator can still leave runs that differ. And the reason is floating-point arithmetic rather than randomness. GPU kernels parallelise reductions across thousands of threads. And the order in which partial sums mix varies between runs. Floating-point addition is not associative — (a + b) + c can differ from a + (b + c) in the last bits — so identical inputs can give slightly other outputs.
Those gaps are around 10−7. But training amplifies them: a tiny gradient gap changes the next weights. It changes the next gradient. And after thousands of steps the runs have visibly diverged. Frameworks offer deterministic modes (torch.use_deterministic_algorithms(True)) that force reproducible kernels, often at a real cost in speed.
What often goes wrong
Seeding one library and assuming it covers the rest. Python, NumPy and the framework each need seeding, plus PYTHONHASHSEED if set ordering matters.
Treating the seed as a hyperparameter. Searching for a good seed is fitting to noise. If results depend heavily on it, the finding is fragile, not tuned.
Reporting a single run. A one-run comparison between two ways is uninterpretable when seed variance alone can move accuracy a point or more. Report mean and spread over a few seeds.
Expecting reproducibility across machines. A fixed seed reproduces a run on the same hardware, library versions and thread count. Change the GPU or the cuDNN version and results move again.
Key takeaway
Training draws randomness from initialisation, shuffling, dropout and augmentation, each from its own generator. So reproducibility means seeding all of them rather than one. Still then GPU reductions are non-deterministic at the level of floating-point rounding. And training amplifies those gaps over thousands of steps. Seed to make a run repeatable for debugging — and report results over a few seeds, since a single run tells you as much about the seed as about the way.
Seed variance is a measurement, not a nuisance
Here is the more useful framing: instead of trying to remove run-to-run variation, measure it.
Train the same setup with five other seeds and record the spread. On many tasks that spread is one or two percent — which means any improvement less than that is indistinguishable from luck.
That has a direct consequence for how results are reported. A single run showing 91.2% against a baseline's 90.8% shows nothing if the seed variance is ±0.8%. Reporting the mean and standard deviation over a few seeds is the honest option. And it is what stops a project from chasing noise for weeks.
It also changes how you choose: a setup that is slightly worse on average but far more stable across seeds is often the better one to ship.
Determinism at inference
Serving has its own reproducibility requirements, and they are often easier.
Call net.eval(). This disables dropout and switches batch normalisation to its running statistics. Without it, predictions vary between calls and depend on what else is in the batch — the same input genuinely gives other answers.
Fix the preprocessing. Ship the fitted scaler, the tokeniser and the exact resizing and normalisation constants alongside the weights. A mismatch here is the most common cause of a net that scored well in training and behaves oddly in production.
Beware batch effects. Still in eval mode, some steps give marginally other results depending on batch composition due to floating-point summation order. For most applications the gap is irrelevant. For anything audited, it is worth knowing about.
Pin the runtime. A net exported to ONNX or TorchScript and run on a other runtime version can differ in the last decimal places.
The same code, twice, with other answers
Five split sources of randomness sit between your script and your weights. Each is showed and then pinned down, in the order they bite.
example_01.pyNumPy
import numpy as np
print("1. INITIALISATION -- the most obvious source.")
for seed in (None, None):
r = np.random.default_rng()
print(" unseeded weights:", np.round(r.normal(size=3), 6))
print(" two runs, two answers. now with a seed:")
for _ in range(2):
print(" seeded weights :", np.round(np.random.default_rng(42).normal(size=3), 6))
print()
print("2. DATA ORDER -- shuffling changes which rows land in which batch.")
data = np.arange(10)
for label, r in (("unseeded", np.random.default_rng()),
("unseeded", np.random.default_rng()),
("seed 0 ", np.random.default_rng(0)),
("seed 0 ", np.random.default_rng(0))):
print(" %s first batch: %s" % (label, r.permutation(data)[:5]))
print()
print("3. DROPOUT / AUGMENTATION -- one extra random draw shifts everything.")
r1, r2 = np.random.default_rng(7), np.random.default_rng(7)
print(" identical seeds stay in step:")
print(" ", (r1.random(6) > 0.5).astype(int), (r2.random(6) > 0.5).astype(int))
r1.random(1)
print(" one extra draw from the first, and they diverge forever:")
print(" ", (r1.random(6) > 0.5).astype(int), (r2.random(6) > 0.5).astype(int))
Output
Floating-point order is the source that surprises people, since addition is not associative in finite precision.
example_02.pyNumPy
import numpy as np
print("4. FLOATING POINT ORDER -- addition is not associative:")
a = np.array([1e16, 1.0, -1e16, 1.0])
print(" left to right : %.4f" % (((a[0] + a[1]) + a[2]) + a[3]))
print(" regrouped : %.4f" % ((a[0] + a[2]) + (a[1] + a[3])))
print(" difference : %.1f"
% abs(((a[0]+a[1])+a[2])+a[3] - ((a[0]+a[2])+(a[1]+a[3]))))
print()
big = np.random.default_rng(0).normal(size=100_000).astype(np.float32)
print(" at scale with float32:")
print(" np.sum : %.8f" % big.sum())
print(" pairwise : %.8f" % np.add.reduce(big))
print(" python sum : %.8f" % float(sum(big.tolist())))
print(" a GPU combines reductions in whatever order threads finish,")
print(" so the same GPU can give two different answers.")
print()
print("5. LIBRARY VERSIONS -- a different BLAS, cuDNN or numpy release")
print(" changes summation order without changing your code.")
print(" this source you cannot fix with a seed, only with a lockfile.")
Output
Given all those sources, here is what to do in practice. And why statistical reproducibility matters more than bit-exact.
example_03.pyNumPy
import numpy as np
print("what to do, in order of how much it buys you:")
print()
print(" # seed every generator, not just one")
print(" random.seed(S); np.random.seed(S)")
print(" torch.manual_seed(S); torch.cuda.manual_seed_all(S)")
print()
print(" # make dataloader workers deterministic")
print(" DataLoader(..., worker_init_fn=seed_worker, generator=g)")
print()
print(" # force deterministic kernels (costs 10-30% speed)")
print(" torch.use_deterministic_algorithms(True)")
print(" torch.backends.cudnn.benchmark = False")
print()
print("the more useful target is STATISTICAL reproducibility:")
print("run with five seeds and report mean and spread.")
rng = np.random.default_rng(1)
scores = 0.842 + rng.normal(0, 0.011, 5)
print(" five seeds: %s" % np.round(scores, 4))
print(" mean %.4f, sd %.4f, range %.4f"
% (scores.mean(), scores.std(), scores.max() - scores.min()))
print(" an 'improvement' of %.3f over a baseline is indistinguishable"
% scores.std())
print(" from seed noise on this model.")
Output
Questions people ask
Why is 42 the conventional seed? Cultural, from The Hitchhiker's Guide to the Galaxy. Any fixed value works. The point is that it is fixed.
Does setting a seed guarantee identical results? On the same machine with the same library versions and deterministic algorithms enabled, yes. Across other hardware, not necessarily.
How much does deterministic mode cost? Often 10–20% throughput, mostly from disabling cudnn.benchmark. Some steps have no deterministic build and will raise an error instead.
Should I always train deterministically? No — use it when debugging or when a result must be just reproducible. For ordinary training, seed all and accept small variation.
How many seeds should I run? Three to five for a paper or an key decision. One is fine for iteration, as long as you do not over-interpret small gaps.
Why do my results differ between my laptop and the server? Other hardware, other library versions, other counts of data loader workers. All three change the arithmetic or the batch composition.
Recap in one screen
Five sources of randomness: initialisation, shuffling, dropout, augmentation and GPU kernel order.
Seed Python, NumPy and Torch, and seed the data loader workers individually.
Full determinism needs cudnn.deterministic and no benchmarking, at 10–20% of throughput.
Reproducibility also needs pinned library versions, versioned data and logged setup.
Measure the seed variance. And treat any improvement less than it as noise.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
What does this module say about “Where the randomness comes from”?
What does this module say about “Seeding, and what a seed really fixes”?
Cheat sheet
Model Reproducibility
Fix the random seed and see exactly what becomes repeatable: initial weights and the order training data is shuffled in.
DEEP LEARNING · vizlearn.in/deep_learning/reproducibility_of_model.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.