Overfitting and out-of-sample evaluation

Overfitting adapts a rule too closely to chance patterns in observed data. Trying many windows, weights or models and selecting the best on the same history makes the reported result depend on that selection. Retain attempted settings and changes made after inspecting validation results.

Time-series evaluation trains on earlier data and evaluates later periods. Forward-return labels crossing split boundaries may require purging boundary observations. Training-fit predictions do not establish performance on unseen data.

The reference course trains on 2017–2019, validates on 2020 and uses 2021–2022 for historical diagnostics, purging by label end dates. Reusing observed history does not demonstrate fresh, unseen OOS performance.

Trying 100 windows or weights on one historical period can produce settings that match chance patterns. Reporting only the best hides the search and selection process. Record the number of trials, selection rules and already inspected periods.

Chronological splits

Training fits a model; validation informs choices; OOS evaluation uses data excluded from training and selection. Changing settings after inspecting evaluation results makes that history observed.

A late-December feature with a five-session forward-return label can end in the next split. Purging removes observations whose label intervals cross the boundary. Sorting rows chronologically alone does not prevent this leakage.

Read the screenshot-led tutorial · All basic concepts · Research methodology

The Probability of Backtest Overfitting — Bailey et al.