Part 2: In-sample and out-of-sample · Chapter 5 of 14 · 9 min
In-sample and out-of-sample
We split the data in time: the first part (65%) to choose, the second (35%) to test. The rule chosen in the first part makes +0.82 ± 0.52 R (55 trades) and in the second +0.43 ± 0.73 (25): it loses half, and stays positive. The first decile of rules goes from +0.38 to +0.10 R, the last from −0.26 to −0.12: the advantage of the first part is kept only in part. On random series the best of the first part (+0.35) is worth +0.02 in the second.
In-sample and out-of-sample
The data on which you choose the rule are the sample, or in-sample data. The data you did not use to choose are out-of-sample. An in-sample result is always optimistic, because the rule was chosen precisely because it did well on those data. An out-of-sample result is honest, because the rule has never seen those data.
The split, for time series, is made in time: the first part to choose, the second to test, in the order in which they happened. Shuffling the days at random would pass information from the future to the past (the volatility of a period, the direction of a stretch of market). And in real life the rule will go on the future, not on a shuffled past.
Here we split the 4,865 candles into 3,162 (until 24 June 2026, 65%) and 1,703 (the rest).
The chosen result and the tested result
We choose, among the 480 rules, the best of the first part (at least 30 trades), and test it on the second:
| First part | Second part | |
|---|---|---|
| Rule | 80-candle breakout, 1.5 ATR stop, 3 R target, duration 48 | the same |
| Trades | 55 | 25 |
| Mean R ± error | +0.82 ± 0.52 | +0.43 ± 0.73 |
| Winning | 40% | |
| Sum | +45.4 R | +10.8 R |
The rule loses about half of the mean R out of sample and stays positive. With 25 trades the error is ±0.73: the value is above zero but the interval contains it. It is neither a confirmation nor a refutation: it is a result that dampens enthusiasm without closing the question.
Read the chart in words
Daily gold chart with, below, the cumulative R curve of the chosen rule and a grey band over the last third of the period, where the curve rises more slowly.
| When | What | Price |
|---|---|---|
| 24 Jun 2026 → 5 Oct 2026 | second part |
Times in UTC.
What happens to all the rules
A single rule says little. Let us look at all 480, first against second part.
| Group of rules | First part | Second part |
|---|---|---|
| The 10 best of the first part | +0.21 | |
| First decile (48 rules) | +0.38 | +0.10 |
| All 480 | −0.01 | −0.02 |
| Last decile | −0.26 | −0.12 |
| Breakout (240 rules) | +0.13 | +0.03 |
| Fade (240 rules) | −0.16 | −0.08 |
(Mean R per rule.) The best rules of the first part stay better than average, but their advantage shrinks a lot: the first decile keeps about a quarter of its +0.38 (+0.10 remains); the last decile, which was −0.26, returns to −0.12. It is the regression to the mean: part of the exceptional result was luck, and luck does not repeat. It is not a strange phenomenon: if you always take "the best" of many, yesterday's best is on average better than the true one and worse than tomorrow's.
Read the chart in words
Daily gold chart with, below, two curves of mean cumulative R per rule: that of the first decile rises a lot up to the split and then almost stops, that of all rules is near zero.
| When | What | Price |
|---|---|---|
| 24 Jun 2026 → 5 Oct 2026 | second part |
Times in UTC.
Some counts confirm it. The rules in profit in the first part are 188 out of 480; of these 104 stay in profit in the second (55%). The rules with a lower limit above zero in the first part are 46; in the second 30 of them are still positive.
There is persistence, but less than it seems
The rank correlation between the mean R of the first and of the second part, over the 480 rules, is +0.43. On random series the same correlation is on average +0.17 (5th percentile −0.40, 95th +0.62): series with no edge have it different from zero because the rules resemble one another, and it varies enormously from series to series. The real +0.43 lies within the range of the random series. It says that something is kept, not how much.
On random series the best of the first part is worth on average +0.35 and in the second +0.02 (5th percentile −0.33, 95th +0.39): in the second part it is positive in 55% of cases, but barely. The difference from the real data (+0.82 → +0.43) is there, in favour of the real data. It is not enough to say there is an edge, but it is the reason why the following chapters do not stop here.
Let us also do the reverse test: take the best of the second part (80-candle breakout, 1 ATR stop, 3 R target, duration 48: 32 trades, +0.48 ± 0.68) and apply it to the first part: +0.43 ± 0.43 over 77 trades. The region of rules around N = 80 is positive in both halves, and the two choices are relatives.
How to split well
- Decide the split before looking at the results, in time order.
- The test part is looked at only once. If you look at it, change the rule and look again, it has become a choice part.
- A test part of 25 trades says little: its error is ±0.7. More trades are needed to tell an edge (book 11, chapter 12).
- Report the two numbers together: "+0.82 in the choice, +0.43 in the test".
- Do not choose the rule by looking at the test: the test part must influence nothing.
Procedure in four steps
- Fix in advance the date that splits the data into a choice part and a test part.
- Choose the rule only on the first part.
- Test it once on the second and report trades, mean R and error.
- If the test contradicts the choice, do not touch the rule and redo the test: either you accept the result or you collect new data.
Exercises
Chapter exercises
1. The first decile of the rules makes +0.38 R in the first part and +0.10 R in the second. What share of the advantage does it keep?
2. Why is the series split in time order and not at random?
3. The chosen rule makes +0.82 ± 0.52 R in the first part and +0.43 ± 0.73 R in the second (25 trades). How is it read?
4. You looked at the second part, tweaked a parameter and tried again. What happened to the second part?
Have a question or want to share your exercise?
Post in the community, or join the free signals room on Telegram.
Educational content, not financial advice. Trading involves risk.