Models & methods
Why one model never wins
A model that dominated last spring can lose in November. Series change, drivers change, and luck exists. Keep a challenger on the same folds, promote on evidence, and stay loyal to the score — not to the architecture that won a bake-off in March.
Updated Aug 6, 2026·7 min read
What happened in March
Someone ran a bake-off. A booster won by a point of WAPE. It got a name, a wiki page, and a year of loyalty. Then November arrived with a different promo mix and a warehouse that had moved. The booster did not get the memo. The error did.
That story is not a reason to distrust models. It is a reason to stop treating a champion as a mascot.
Same folds, five candidates
Seasonal naive
16.4%
Smoothing
14.1%
Gradient boost
11.2%
Foundation
9.7%
Fine-tuned
10.1%
Why models race
Champion–challenger: one model is live. Others train on the same data and the same folds. If a challenger wins by enough, it takes the job. Ensembles (a median of uncorrelated decent models) are the softer version. Both beat an annual “model refresh” project that ships late on purpose.
- Same grain, horizon, and interval after a switch — the contract stays.
- Tell people when the driver mix changes. They do not need the architecture name.
- Keep a naive watchdog so you notice death early.
Switching without drama
Automate the promotion with a threshold, not a steering committee. Humans still need a pin for regulated lines or a freeze during a known mess. The tournament can keep running in the dark so you know what you are refusing.
Where a human still sits
Overrides, scenario design, “this week the feed is a lie.” The human should not be refitting ARIMA by hand because the last bake-off felt personal. Loyalty to the score is the adult move. Loyalty to last March’s winner is how forecasts go stale with a nice title.
FAQ
- What is a champion–challenger setup?
- The champion is live. Challengers train on the same data and the same folds. If a challenger wins by enough, it becomes the champion. You do not wait for an annual 'model refresh' project.
- Won't switching models confuse the business?
- Switching the engine should not switch the contract. Same grain, same horizon, same interval. What changes is error. Tell people when the driver mix changes; they do not need the architecture name.
- Can I lock a model on purpose?
- Yes — regulated lines, or a period you refuse to let an algorithm roam. A tournament with a manual pin is still better than a forgotten notebook.
Keep going
Guide
Forecasting models compared
Classical models are interpretable and cheap. Machine learning eats more drivers. Foundation models arrive pretrained and can go zero-shot. None of them win every series — which is why serious stacks keep a tournament, not a favorite.
Guide
Why forecasts go stale in production
A forecast starts dying the moment the world moves and the model does not. New promotions, a competitor, a warehouse fire, a quiet change in how you record demand — live systems need a refresh cadence, drift checks, and a way to re-score the champion.
Guide
How to backtest a forecasting model
Backtesting asks the model to forecast a past window it was not trained on, then scores the miss. Walk the window forward so you see many 'futures,' not one lucky test month.
Use case
Revenue forecasting
Maintain a current revenue outlook grounded in bookings, pipeline, usage, renewals, pricing, and seasonality.
Use case
Retail demand forecasting
Anticipate demand by product, store, channel, and region before buying or allocation decisions are locked.