What Is Out-of-Sample Evaluation? Handling Periods You Have Already Inspected

Out-of-sample evaluation tests rules on data not used to create or select them. It asks whether rules adjusted to historical results behave similarly on different data.

A favorable result does not prove future success. It does, however, provide evidence beyond performance on the development sample when examining overfitting.

For illustration, split time chronologically into A, B and C. Develop rules in A, compare candidates in B and reserve C for a final evaluation. These are fictional segments, not actual research windows.

If you change conditions after examining B, B has contributed to selection. Even without overlapping A, it no longer has the same meaning as an untouched final evaluation period. Record how each segment was used instead of relying on its label.

Adding a condition because a particular period performed poorly brings information from that period into the revised rules. Even without changing an equation, using results to accept or reject a candidate contributes to selection.

A repeatedly inspected period can be described as a separate diagnostic period. It should not be presented as an untouched new evaluation. Inspection history and the number of candidates considered help interpret the result.

Chronological splitting alone does not eliminate look-ahead bias. Check whether features or normalization use evaluation-period information and whether forward-return windows cross a boundary.

For trained models, record which data fitted preprocessing. A feature date may lie in the development segment while part of its future-return label lies in the next segment. Decide how to handle such labels as part of the evaluation design.

Keep inputs, rules, fees, execution assumptions, period roles and run time together. Record why a change was made and which results had already been inspected.

KabuForge associates fixed inputs and settings with run records. Freezing files alone does not prove that an evaluation period remained unseen. Software reproducibility and independence of research evaluation are separate checks.

Inspect universe size, missing data, market conditions and execution assumptions when comparing periods. A favorable out-of-sample result remains limited if costs or information timing are incorrect.

The portfolio and models tutorial introduces the public course’s period-splitting workflow. Replaying that historical course is different from observing new market data as it arrives.

KabuForge GUI research courses.

This article teaches research procedures. It does not recommend trades or guarantee future returns.Disclaimer

Bailey et al. on backtest overfitting。 The original paper studies selection among repeated backtests; a simple split does not remove all selection effects.

Explore the backtesting learning path

• What Daily Bars Cannot Tell You: Highs, Lows and Execution Order
• What is turnover? Distinguish one-sided and two-sided measures
• What is slippage? Separate reference prices from fills