Accuracy & evaluation

Backtesting

Backtesting scores a forecast on a past window the model was not trained on. Done properly, you walk that window forward so you see many futures, not one lucky test month.

A single holdout is a story. Four folds that all sit in Q4 are a holiday story. Spread the origins across the year, keep the horizon equal to production, and average the pain.

The point is not to torture the data scientist. The point is to estimate live error before you let the number into replenishment. If backtest WAPE is 11% and live is 24%, something leaked or the world moved. Find which.

Refit cadence belongs in the backtest. A model you would retrain weekly should not be scored as if it were fit once in January.