Simulate the passage of time. Watch how a perfectly trained model degrades in production as the underlying data distribution slowly drifts away from the training baseline.
Drift is a change in the data a deployed model receives, the outcomes it predicts, or the relationship between them. Monitoring can reveal changes that deserve investigation, but a shift alone does not prove model performance has fallen.
P(Y | X) changes. The same inputs now lead to different outcomes, perhaps because customer behavior or policy changed.In practice, these changes can occur together. A model’s measured performance may also fall because of data-quality, software, or serving changes. Drift is a signal to investigate, not a diagnosis or automatic command to retrain.
Select Concept Drift in the visualizer and use Simulate Time. The underlying relationship changes over time while the deployed model stays fixed, so its error grows. Use Retrain model to compare how fitting to newer data changes the result. The demonstration is deliberately simple: in a real system, retraining helps only if recent labels are available, representative, and reliable.
Log the model version, prediction time, inputs allowed by policy, prediction, and eventual outcome. Compare input and score distributions with a reference period to spot changes early; these checks need no labels, but cannot tell whether accuracy changed. When labels arrive, monitor the task metric and confusion matrix over time, including important subgroups.
Define investigation thresholds from the system’s normal variation and business risk rather than treating one statistic as a universal alarm. Check data collection, feature pipelines, class prevalence, calibration, and performance before retraining. Record the training window and evaluation results so a new model can be compared with the deployed one and rolled back if needed.
About to run: set Time (Months) to its maximum (24). Before it does — what happens to the readout?
Committing to an answer first is the point — the reveal runs the experiment on the visualisation above and reads the real value back, so nothing here is scripted.
Say the answer out loud before you reveal it — recalling it is what makes it stick, and rereading it is not.
What does this module say about “Three kinds of change”?
What does this module say about “See the relationship move”?
What does this module say about “Sources and next steps”?
Fast-forward through production and watch a well-trained model decay as incoming data drifts away from what it learned.