True reproducibility means fixing the Random Seed controls both the initial network weights and the order in which data is shuffled during training. Change the seed to see the data sequence entirely rearrange, then switch back to Seed 42 to verify identical structural recovery!
Overview
Where the randomness comes from
Neural network training is randomised in at least four independent places, and every one of them changes the final weights:
Weight initialisation. Every weight starts as a random draw. Different draws land the optimiser in different basins of the loss surface.
Data shuffling. Mini-batches are formed from a shuffled dataset, so a different order means different gradients at every step.
Dropout. Which units are zeroed is resampled on every forward pass during training.
Augmentation. Random crops, flips and noise mean the model literally never sees the same input twice.
None of these is a defect — each is deliberately there to improve generalisation. But they mean “the model” is a sample from a distribution of models, not a fixed object.
Determinism Checks
Active Seed Environment
SEED: 42
Determinized Data Pipeline
Sample #0
The Seed controls the random shuffle order.
Weight Hash
--
Sum of pseudo-random weights.
Activation Hash
--
Dependent on specific data sample.
PRO TIP: Memorize the exact data order (D1..D12) for Seed 42. Change the seed to scramble the order, then change it back to verify identical shuffling!
Drag to Pan | Scroll to Zoom
Model Reproducibility: A Practical Guide
Train the same model twice on the same data and you get two different results. Controlling the randomness is what turns a demo into an experiment you can trust.
Seeding, and what a seed actually fixes
These are all pseudo-random: a deterministic sequence generated from a starting value called the seed. Fix the seed and the sequence is identical every run.
random.seed(42) # Python’s own generator
np.random.seed(42) # NumPy
torch.manual_seed(42) # PyTorch, CPU and GPU
The catch is that these are separate generators. Seeding NumPy does nothing for PyTorch’s dropout, and seeding PyTorch does nothing for a shuffle done with Python’s random. Every library that draws a random number needs its own seed, and missing one is the usual reason a “seeded” run is still not reproducible.
Why the same code gives different numbers
Run the same training script twice and the results differ. Five sources of randomness are involved, and each has to be pinned separately.
Source
What varies
Weight initialisation
The starting point
Data shuffling
The order and composition of batches
Dropout
Which units are zeroed each step
Augmentation
The random transformations applied
GPU kernels
Floating-point summation order
The first four are ordinary pseudo-random number generators and can be seeded. The fifth is different in kind: many GPU operations sum values in a non-deterministic order, and floating-point addition is not associative, so (a + b) + c and a + (b + c) can differ in the last bits. Those differences amplify over thousands of steps.
Seeding gets you most of the way; full determinism costs a little speed.
Pinning it down
import os, random, numpy as np, torch
def set_seed(seed=42):
os.environ["PYTHONHASHSEED"] = str(seed)
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
set_seed(42)
# for full determinism, at some cost in speed
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.use_deterministic_algorithms(True)
Two additional details that are easy to miss.
Data loader workers need their own seeding. Each worker process inherits a different base seed, and without worker_init_fn several workers can generate identical augmentation sequences — or different ones between runs.
cudnn.benchmark = True is non-deterministic by design. It profiles several convolution algorithms and picks the fastest, which can differ between runs. Leaving it on is often worth 10–20% speed; turning it off is required for bit-exact reproducibility.
Reproducibility is more than the seed
A fixed seed makes one script repeatable on one machine. Reproducing a result months later, or on someone else's machine, needs more.
Pin the environment. Library versions change numerical behaviour and occasionally defaults. Record exact versions — requirements.txt with pinned versions, a lock file, or a container image.
Version the data. "The dataset" is a moving target if it is a folder someone adds files to. Record a hash, a snapshot, or use a data-versioning tool.
Log the configuration. Every hyperparameter, the seed, the git commit, and the command line. A result you cannot map back to its settings is not a result.
Save checkpoints with their config. A weights file alone does not tell you what preprocessing it expects.
Tools such as MLflow, Weights & Biases and DVC exist to automate this, and a disciplined text log plus git tags covers most of it.
Interactive Exploration Guide
Fix the seed and repeat. Enter a seed value and press Apply Seed & Init, note the initial weights, then press it again with the same seed. Identical values — the initialisation is fully determined by that number.
Change one digit. Change the seed and re-initialise. Every weight is different. The seed is not a setting with a meaningful scale; it just selects an arbitrary point in the sequence.
Watch the sequence advance. Press Next Sample several times. Each draw differs, but the whole run of draws replays identically after re-applying the same seed — that is what pseudo-random means.
Stream without reseeding. Press Auto Stream and let it run, then re-apply the seed. The stream restarts from the beginning of the same sequence, not from where it stopped.
Why the GPU can still be non-deterministic
Seeding every generator can still leave runs that differ, and the reason is floating-point arithmetic rather than randomness. GPU kernels parallelise reductions across thousands of threads, and the order in which partial sums combine varies between runs. Floating-point addition is not associative — (a + b) + c can differ from a + (b + c) in the last bits — so identical inputs can give slightly different outputs.
Those differences are around 10−7, but training amplifies them: a tiny gradient difference changes the next weights, which changes the next gradient, and after thousands of steps the runs have visibly diverged. Frameworks offer deterministic modes (torch.use_deterministic_algorithms(True)) that force reproducible kernels, typically at a real cost in speed.
What usually goes wrong
Seeding one library and assuming it covers the rest. Python, NumPy and the framework each need seeding, plus PYTHONHASHSEED if set ordering matters.
Treating the seed as a hyperparameter. Searching for a good seed is fitting to noise. If results depend heavily on it, the finding is fragile, not tuned.
Reporting a single run. A one-run comparison between two methods is uninterpretable when seed variance alone can move accuracy a point or more. Report mean and spread over several seeds.
Expecting reproducibility across machines. A fixed seed reproduces a run on the same hardware, library versions and thread count. Change the GPU or the cuDNN version and results move again.
Key takeaway
Training draws randomness from initialisation, shuffling, dropout and augmentation, each from its own generator, so reproducibility means seeding all of them rather than one. Even then GPU reductions are non-deterministic at the level of floating-point rounding, and training amplifies those differences over thousands of steps. Seed to make a run repeatable for debugging — and report results over several seeds, because a single run tells you as much about the seed as about the method.
Seed variance is a measurement, not a nuisance
Here is the more useful framing: instead of trying to eliminate run-to-run variation, measure it.
Train the same configuration with five different seeds and record the spread. On many tasks that spread is one or two percent — which means any improvement smaller than that is indistinguishable from luck.
That has a direct consequence for how results are reported. A single run showing 91.2% against a baseline's 90.8% demonstrates nothing if the seed variance is ±0.8%. Reporting the mean and standard deviation over several seeds is the honest alternative, and it is what prevents a project from chasing noise for weeks.
It also changes how you choose: a configuration that is slightly worse on average but far more stable across seeds is often the better one to ship.
Determinism at inference
Serving has its own reproducibility requirements, and they are usually easier.
Call model.eval(). This disables dropout and switches batch normalisation to its running statistics. Without it, predictions vary between calls and depend on what else is in the batch — the same input genuinely gives different answers.
Fix the preprocessing. Ship the fitted scaler, the tokeniser and the exact resizing and normalisation constants alongside the weights. A mismatch here is the most common cause of a model that scored well in training and behaves oddly in production.
Beware batch effects. Even in eval mode, some operations produce marginally different results depending on batch composition due to floating-point summation order. For most applications the difference is irrelevant; for anything audited, it is worth knowing about.
Pin the runtime. A model exported to ONNX or TorchScript and run on a different runtime version can differ in the last decimal places.
The same code, twice, with different answers
Five separate sources of randomness sit between your script and your weights. Each is demonstrated and then pinned down, in the order they bite.
example_01.pyNumPy
import numpy as np
print("1. INITIALISATION -- the most obvious one.")
for seed in (None, None):
r = np.random.default_rng()
print(" unseeded weights:", np.round(r.normal(size=3), 6))
print(" two runs, two answers. now with a seed:")
for _ in range(2):
print(" seeded weights :", np.round(np.random.default_rng(42).normal(size=3), 6))
print()
print("2. DATA ORDER -- shuffling changes which rows land in which batch,")
print(" which changes every gradient after the first.")
data = np.arange(10)
for label, r in (("unseeded", np.random.default_rng()),
("unseeded", np.random.default_rng()),
("seed 0 ", np.random.default_rng(0)),
("seed 0 ", np.random.default_rng(0))):
print(" %s first batch: %s" % (label, r.permutation(data)[:5]))
print()
print("3. DROPOUT AND AUGMENTATION -- more draws, usually from a different")
print(" generator than the one you seeded.")
r1, r2 = np.random.default_rng(7), np.random.default_rng(7)
print(" two generators seeded identically stay in step:")
print(" ", (r1.random(6) > 0.5).astype(int), (r2.random(6) > 0.5).astype(int))
r1.random(1) # one extra draw from r1 only
print(" one extra draw from the first, and they diverge forever:")
print(" ", (r1.random(6) > 0.5).astype(int), (r2.random(6) > 0.5).astype(int))
print(" this is why a seed set once at the top is not enough. any code")
print(" path that consumes a different NUMBER of random values -- a")
print(" conditional augmentation, an early break -- shifts everything")
print(" downstream.")
print()
print("4. FLOATING POINT ORDER -- the one that surprises people. addition")
print(" is not associative in floating point:")
a = np.array([1e16, 1.0, -1e16, 1.0])
print(" the same four numbers, summed in two orders:")
print(" left to right : %.4f" % (((a[0] + a[1]) + a[2]) + a[3]))
print(" regrouped : %.4f" % ((a[0] + a[2]) + (a[1] + a[3])))
print(" both are 'correct'. neither is wrong. they differ by %.1f."
% abs(((a[0] + a[1]) + a[2]) + a[3] - ((a[0] + a[2]) + (a[1] + a[3]))))
print()
big = np.random.default_rng(0).normal(size=100_000).astype(np.float32)
print(" and at scale, with float32, the effect is routine:")
print(" np.sum : %.8f" % big.sum())
print(" pairwise : %.8f" % np.add.reduce(big))
print(" python sum : %.8f" % float(sum(big.tolist())))
print(" a GPU splits a reduction across thousands of threads and combines")
print(" them in whatever order they finish. that order is not guaranteed")
print(" to repeat, which is why the same GPU can give two answers.")
print()
print("5. LIBRARY VERSIONS -- a different BLAS, cuDNN algorithm or numpy")
print(" release changes the summation order without changing your code.")
print(" this is the source you cannot fix with a seed, only with a lockfile.")
print()
print("what actually to do, in order of how much it buys you:")
print()
print(" # every generator, not just one")
print(" random.seed(S); np.random.seed(S)")
print(" torch.manual_seed(S); torch.cuda.manual_seed_all(S)")
print()
print(" # make the dataloader workers deterministic too")
print(" DataLoader(..., worker_init_fn=seed_worker, generator=g)")
print()
print(" # force deterministic kernels, and accept the slowdown")
print(" torch.use_deterministic_algorithms(True)")
print(" torch.backends.cudnn.benchmark = False")
print()
print("that last pair is a real cost -- cudnn.benchmark picks the fastest")
print("algorithm for your shapes at runtime, and turning it off can cost 10")
print("to 30 percent. bit-exact reproducibility is a thing you buy, not a")
print("thing you get.")
print()
print("the more useful target is usually STATISTICAL reproducibility: run")
print("with five seeds and report mean and spread. a result that only holds")
print("for seed 42 is not a result, and a paper that reports one number")
print("without a spread has not told you whether its improvement is real:")
rng = np.random.default_rng(1)
scores = 0.842 + rng.normal(0, 0.011, 5)
print(" five seeds: %s" % np.round(scores, 4))
print(" mean %.4f, sd %.4f, range %.4f"
% (scores.mean(), scores.std(), scores.max() - scores.min()))
print(" an 'improvement' of %.3f over a baseline would be indistinguishable"
% (scores.std()))
print(" from seed noise on this model.")
Output
Questions people ask
Why is 42 the conventional seed? Cultural, from The Hitchhiker's Guide to the Galaxy. Any fixed value works; the point is that it is fixed.
Does setting a seed guarantee identical results? On the same machine with the same library versions and deterministic algorithms enabled, yes. Across different hardware, not necessarily.
How much does deterministic mode cost? Typically 10–20% throughput, mostly from disabling cudnn.benchmark. Some operations have no deterministic implementation and will raise an error instead.
Should I always train deterministically? No — use it when debugging or when a result must be exactly reproducible. For ordinary training, seed everything and accept small variation.
How many seeds should I run? Three to five for a paper or an important decision. One is fine for iteration, as long as you do not over-interpret small differences.
Why do my results differ between my laptop and the server? Different hardware, different library versions, different numbers of data loader workers. All three change the arithmetic or the batch composition.
Recap in one screen
Five sources of randomness: initialisation, shuffling, dropout, augmentation and GPU kernel order.
Seed Python, NumPy and Torch, and seed the data loader workers individually.
Full determinism needs cudnn.deterministic and no benchmarking, at 10–20% of throughput.
Reproducibility also requires pinned library versions, versioned data and logged configuration.
Measure the seed variance, and treat any improvement smaller than it as noise.
Recall check
0 of 3
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
Without scrolling back — what is the one-line takeaway from this module?
Training draws randomness from initialisation, shuffling, dropout and augmentation, each from its own generator, so reproducibility means seeding all of them rather than one. Even then GPU reductions are non-deterministic at the level of floating-point rounding, and training amplifies those differences over thousands of steps.
What does this module say about “Where the randomness comes from”?
Neural network training is randomised in at least four independent places, and every one of them changes the final weights:
What does this module say about “Seeding, and what a seed actually fixes”?
These are all pseudo -random: a deterministic sequence generated from a starting value called the seed. Fix the seed and the sequence is identical every run.
Cheat sheet
Model Reproducibility
True reproducibility means fixing the Random Seed controls both the initial network weights and the order in which data is shuffled during training. Change the seed to see the data sequence entirely rearrange, then switch back to Seed 42 to verify identical structural recovery!
DEEP LEARNING · vizlearn.in/deep_learning/reproducibility_of_model.html
Ashish Jangra builds and maintains VizLearn. Every module here is written and the visualisation behind it hand-built, so the numbers in a readout come from the same code that draws the picture. Corrections are genuinely welcome and get priority over everything else — if a page states something wrong, or an animation misrepresents what the algorithm does, get in touch.