In production
Why forecasts go stale in production
A forecast starts dying the moment the world moves and the model does not. New promotions, a competitor, a warehouse fire, a quiet change in how you record demand — live systems need a refresh cadence, drift checks, and a way to re-score the champion.
Updated Aug 23, 2026·8 min read
Two clocks
A forecast is trained on a window. The world keeps a different calendar. Every day those clocks disagree a little. After a few weeks they can disagree enough to empty a shelf or fill a cooler with product nobody asked for. Staleness is that gap, not a moral failing of the last data scientist.
The world moved. The model did not.
The kinds of drift
Data drift is the inputs looking different: a new POS, a promo feed that silently stopped, SKU IDs remapped in March. Concept drift is the relationship changing: the same flag no longer lifts because the market got used to 10% off. You fix them differently. You should watch for both.
A third kind is boredom: nobody refreshed the champion, the pipeline still “runs,” and the number in the app is last month’s export with a live timestamp. That one is process.
Boring breakage
- Late files padded with zeros that look like a demand miracle.
- Timezone shifts after a vendor migration.
- A feature job that skipped holidays in the new region.
- Inference pointing at a model pin from a freeze everyone forgot.
The first runbook step for a crazy forecast is not “retrain.” It is “is the pipe alive, and is this the model we think it is?”
Keeping it live
Refresh at least as often as the decisions you feed. Daily ops want a daily rebuild or a rolling update. A quarterly S&OP number can refresh weekly and still needs a drift alarm. Keep a naive baseline in production. Re-score the tournament on a schedule. Alert when actuals leave the band, not when a human happens to open the dashboard.
FAQ
- Why do forecasts get worse over time?
- The world changes and the training window does not. Demand shifts, a driver disappears, or the data feed silently pads zeros. Error creeps up until someone notices in a monthly review — which is late.
- How often should a forecast refresh?
- At least as often as the decisions it feeds. Daily ops want a daily rebuild or a rolling update. A quarterly S&OP number can refresh weekly, but it still needs a drift alarm.
- What is the difference between data drift and concept drift?
- Data drift is the inputs looking different (a new POS, a missing promo feed). Concept drift is the relationship changing (the same promo no longer lifts). You fix them differently. You should watch for both.
Keep going
Guide
Anomaly detection for business metrics
A fixed threshold treats every Tuesday like every Saturday. A forecast band asks whether today is weird given what the series usually does at this clock. That is the difference between an on-call fire drill and a useful alert.
Guide
Why one model never wins
A model that dominated last spring can lose in November. Series change, drivers change, and luck exists. Keep a challenger on the same folds, promote on evidence, and stay loyal to the score — not to the architecture that won a bake-off in March.
Guide
How to backtest a forecasting model
Backtesting asks the model to forecast a past window it was not trained on, then scores the miss. Walk the window forward so you see many 'futures,' not one lucky test month.
Use case
Retail demand forecasting
Anticipate demand by product, store, channel, and region before buying or allocation decisions are locked.
Use case
Shipment volume forecasting
Forecast inbound and outbound volume by lane, facility, customer, and mode before capacity is committed.