Simulate the passage of time. Watch how a perfectly trained model degrades in production as the underlying data distribution slowly drifts away from the training baseline.
A machine learning model is a snapshot of the world at the moment it was trained. But the world is not static. Customer preferences change, economic conditions shift, and new patterns emerge. Model Drift is the degradation of a model's predictive power over time because the real-world environment has changed since the model was deployed.
This visualization shows a model trained at "Month 0". As you simulate the passage of time, you'll see the relationship between the model's predictions and the live data diverge, causing the error to increase. There are two primary types of drift to explore.
A machine learning model is a snapshot of the world at the moment it was trained. But the world is not static. Customer preferences change, economic conditions shift, and new patterns emerge. Model Drift is the degradation of a model's predictive power over time because the real-world environment has changed since the model was deployed.
This occurs when the fundamental relationship between the input variables and the target variable changes. The "rules of the game" have changed. In the visualization, the green dashed line (the true underlying pattern) will slowly change its shape over time, while the data points continue to follow it. The deployed model (red line), which learned the original pattern, becomes increasingly wrong.
This occurs when the distribution of the input data changes, even if the underlying concept remains the same. The model starts seeing data it has never encountered before. In the visualization, the green dashed line will remain static, but the blue data points will drift horizontally into a new region. The model, which was only trained on data from the initial region, has no idea how to make accurate predictions for these new inputs. A third type, Sudden Shock, is an extreme form of drift where a major event instantly changes the data or concept, like the effect of a global pandemic on shopping behavior.
A trained model is a photograph. It records the relationships that held in the data it saw, on the day it saw it, and then it stops updating — forever, unless you retrain it.
Everything else keeps moving. A demand forecast built before a competitor opened nearby. A fraud model built before fraudsters read about it. A recommendation model built before a new product category existed. None of these models break loudly; they degrade, quietly, while continuing to return confident predictions.
This is why "the model is finished" is never true of a deployed system. A model in production is a component with a maintenance schedule, and drift monitoring is that schedule.
The word "drift" covers three genuinely different problems, and the fixes differ.
Data drift (covariate shift) — the inputs change, the relationship does not. Your customer base shifts younger; the way age relates to purchase behaviour stays the same. The model is now making predictions in a region it saw less of during training, so accuracy typically drops even though nothing about the underlying rule changed.
Concept drift — the relationship itself changes. The inputs might look identical, but what they imply is different. "Working from home" meant something different about a customer in 2019 than in 2021. This is the serious one: no amount of reweighting the old data fixes it, because the old data is now wrong.
Label drift (prior shift) — the mix of outcomes changes. Fraud rises from 0.5% to 2% of transactions. The model's calibration is now off even if its ranking is still good, and a fixed threshold is now in the wrong place.
| Type | What changed | Typical fix |
|---|---|---|
| Data drift | P(X) | Retrain on recent data; reweight |
| Concept drift | P(y given X) | Retrain, and shorten the training window |
| Label drift | P(y) | Recalibrate, adjust the threshold |
Concept drift also comes in speeds. Sudden: a regulation changes overnight. Gradual: preferences shift over a year. Recurring: seasonality, which is not really drift at all if your features include the season.
The awkward reality is that ground-truth labels usually arrive late — you learn whether a loan defaulted a year after you approved it. So monitoring splits into two layers.
Input monitoring works immediately, because it needs no labels:
Outcome monitoring is the real test, and it lags:
Set thresholds and alerts on these before launch, not after the first incident. And log every prediction with its inputs and a timestamp, or you will have nothing to compare against later.
The inputs change, the relationship changes, or the class balance changes. Each one has a different signature, and only one of them is visible without labels.
Use the interactive panel to see how drift destroys a model's performance and how retraining can fix it.
Drift is not a sign of a bad model; it is an inevitability for any model deployed in a dynamic environment. The solution is not to build a "perfect" model but to have a robust MLOps (Machine Learning Operations) strategy. This involves:
Detection is only useful if there is a response attached, and there are three, in increasing order of effort.
Scheduled retraining. The simplest policy: refit on a rolling window every week or month. Cheap to automate, and it handles gradual drift well. The main decision is the window length — too long and it dilutes recent reality, too short and the model becomes noisy and unstable.
Triggered retraining. Retrain when a monitor crosses a threshold rather than on a calendar. More efficient, and it responds to sudden drift far faster. It requires the monitoring to be trustworthy, since a false alarm costs a full retraining cycle.
Redesign. Sometimes the features themselves have stopped being meaningful — a channel was discontinued, a product was renamed, a data source changed its schema. No retraining fixes that; the pipeline needs work.
Two practices make all three safer. Keep a champion–challenger setup, where the newly trained model runs alongside the live one on real traffic before replacing it — a retrained model is not automatically a better model. And keep a small, stable golden test set that never changes, so you can distinguish "the model got worse" from "the new test data got harder".
A workable minimum for any deployed model:
Open-source tools such as Evidently, NannyML and river cover most of this, and every major cloud ML platform has a monitoring product. The tooling matters much less than having decided, before launch, what "too much drift" means for this model.
How often should I retrain? As often as the underlying process changes. Fraud and pricing models often need weekly or daily attention; a model predicting physical properties may run for years. Measure the decay rate rather than guessing: retrain on old data, evaluate on progressively newer windows, and see how fast performance falls.
Can a model drift if the data does not? The model itself does not change, but the world's relationship to it can — and a stable input distribution with falling accuracy is the signature of concept drift.
Is drift the same as an outlier? No. An outlier is one unusual row; drift is a sustained change in the distribution. Both matter, and both need separate detection.
Should I always retrain on the newest data only? Not always. Recency helps with concept drift and hurts when it discards rare but still-valid patterns. Weighted training — recent rows count more, old rows still count — is often a better compromise.
What if I never get labels? Then input drift monitoring and prediction monitoring are all you have, plus any proxy outcome you can obtain: click-throughs, manual review outcomes, complaint rates.
Does drift affect deep learning models more? Not inherently, but they are often deployed on inputs that shift more — user-generated text, camera images, sensor streams — and they are more expensive to retrain, so the monitoring matters more.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “Overview”?
A machine learning model is a snapshot of the world at the moment it was trained. But the world is not static. Customer preferences change, economic conditions shift, and new patterns emerge. Model Drift is the degradation of a model's predictive power over time because the real-world environment has changed since the model was deployed.
What does this module say about “Types of Drift”?
This occurs when the fundamental relationship between the input variables and the target variable changes. The "rules of the game" have changed. In the visualization, the green dashed line (the true underlying pattern) will slowly change its shape over time, while the data points continue to follow it. The deployed model ( red line ), which learned the original pattern, becomes increasingly wrong. 2.
What does this module say about “Concept Drift”?
This occurs when the fundamental relationship between the input variables and the target variable changes. The "rules of the game" have changed. In the visualization, the green dashed line (the true underlying pattern) will slowly change its shape over time, while the data points continue to follow it. The deployed model ( red line ), which learned the original pattern, becomes increasingly wrong. 2.
Simulate the passage of time. Watch how a perfectly trained model degrades in production as the underlying data distribution slowly drifts away from the training baseline.