Accuracy & evaluation
How to backtest a forecasting model
Backtesting asks the model to forecast a past window it was not trained on, then scores the miss. Walk the window forward so you see many 'futures,' not one lucky test month.
Updated Aug 21, 2026·9 min read
Why a single test month lies
One holdout is a story. If that month contained your only heatwave, the model that memorized heat looks like a genius. If it did not, the model that ignores weather looks fine until July. Walk the origin forward. Average the pain.
Walk-forward folds
Walk-forward, pictured
Train on the past. Predict the next horizon — the same length you will serve. Slide the origin. Repeat. Expanding windows keep more history each time. Sliding windows forget a distant regime on purpose. Both are legal. Random k-fold is not.
Match refit cadence. A model you would retrain weekly should not be scored as if it were fit once in January. That gap is how notebooks beat production.
Leakage to hunt
- Rolling stats that include the day you are predicting.
- A covariate you will not have at 6am (tomorrow’s temperature, not this morning’s forecast).
- Target leakage through a slowly-arriving ‘daily sales’ file that actually settles three days later.
- Standardizing with the whole series, including the test window.
How to read the results
Report the same metric the business will feel, versus a naive baseline, with grain and horizon in the sentence. Look at folds individually. Four lovely folds and one disaster is a seasonal hole, not an average to round away.
Plot error against volume. Models that only win on tiny SKUs are winning a trivia contest. Slice promotions on vs off. If the model only wins when nothing happens, you have a rest-day forecaster.
How often to rerun
Whenever you change features, horizon, or the data contract — and on a schedule anyway. Tournaments that run themselves are how you notice a champion dying in week three instead of at the QBR.
FAQ
- What is backtesting in forecasting?
- You hide a slice of history, train on what came before it, forecast that slice, and score the error. Then you slide the slice forward and do it again. The average of those scores is closer to live error than a single holdout.
- What is walk-forward validation?
- A backtest that keeps time's arrow. Train up to t, predict t+1…t+h, then move t forward. You never train on a Tuesday to predict the Monday before it.
- How many folds do I need?
- Enough to cover the seasons you care about. Four folds that all sit in Q4 will bless a holiday-happy model. Spread them, and keep the fold horizon equal to the live horizon.
Keep going
Guide
WAPE, explained
WAPE (weighted absolute percentage error) is total absolute error divided by total actuals. It behaves when some periods are zero, and it does not let tiny SKUs dominate the score the way MAPE does.
Guide
Why forecasts go stale in production
A forecast starts dying the moment the world moves and the model does not. New promotions, a competitor, a warehouse fire, a quiet change in how you record demand — live systems need a refresh cadence, drift checks, and a way to re-score the champion.
Guide
Why one model never wins
A model that dominated last spring can lose in November. Series change, drivers change, and luck exists. Keep a challenger on the same folds, promote on evidence, and stay loyal to the score — not to the architecture that won a bake-off in March.
Use case
Retail demand forecasting
Anticipate demand by product, store, channel, and region before buying or allocation decisions are locked.
Use case
Energy load forecasting
Forecast demand by interval, feeder, zone, and customer class with weather-driven uncertainty.