Part 4: The system under test · Chapter 12 of 14 · 9 min
Robustness and overfitting: how well a result holds
Four robustness tests on the system (a grid of 96 variants, a comparison with randomly assigned filters, two halves of the period, different thresholds): the filters together beat chance in 1-3% of random trials, but the result comes from 25 trades, nearly all supply in the second half of the period.
The problem
A result found after trying many variants is higher than one chosen in advance. If you try a hundred variants of a system with no edge, the best has a positive result. Chapter 11 gave two positive versions (B and C) put together after seeing the data. Here we measure how well the result holds with four tests.
Test 1: the grid
The variants are 96 (16 combinations of the four filters times six orders: 1.5R, 2R or 3R target, stop margin of 0.25 or 0.5 ATR). With at least 20 trades, 54 remain.
| Group | Variants |
|---|---|
| With positive mean R | 28 out of 54 |
| With positive mean R beyond one error | 9 |
| With positive mean R beyond two errors | 1 |
The top five: trend plus half at 3R with a 0.5 ATR margin (25 trades, +0.82 ± 0.39 R), trend plus half at 3R with a 0.25 margin (+0.79 ± 0.40), trend plus take at 3R (23 trades, +0.74 ± 0.39 and +0.55 ± 0.40), trend plus half at 2R with a 0.5 margin (+0.41 ± 0.29). The bottom five run from −0.14 to −0.15 R (no filter and 1.5R, half only or break only, trend plus break). The mean of the variants' mean R is +0.09, the median 0.00.
Test 2: what chance finds
The filters are assigned to POI at random (shuffled across the 291) and the grid is repeated, 1,000 times. The best variant with at least 20 trades has, in this null case, a median mean R of +0.31 (5th percentile +0.06, 95th +0.70). The observed value (+0.82) is exceeded by 3% of the random trials. The mean of the mean R of all the variants is +0.09 against a random median of −0.09 (95th percentile +0.05): the observed mean is exceeded by 1.4% of the random trials.
With randomly assigned filters, then, the best variant of such a wide grid already has a positive R (+0.31 as the median). What stands out is not the highest result, but the fact that the set of variants sits above chance: in this sample the four filters together show a signal beyond chance in 1-3% of trials.
Test 3: two halves of the period
The walk-forward picks the best variant in one half and tests it in the other:
| Choice | Variant chosen | In the choice half | In the other half |
|---|---|---|---|
| First half | Trend plus take, 3R, margin 0.5 | 10 trades, +0.47 ± 0.58 | 13 trades, +0.95 ± 0.54 |
| Second half | Trend plus half, 3R, margin 0.25 | 16 trades, +1.23 ± 0.51 | 9 trades, 0.00 ± 0.58 |
The result is opposite in the two directions, with 9-16 trades and errors of 0.5. Version B in particular: first half (up to 24 May) 9 trades, −0.22 ± 0.44; second half 16 trades, +0.67 ± 0.38. By side: demand 4 trades (−0.26 ± 0.75), supply 21 (+0.46 ± 0.33). Version E: −0.21 ± 0.44 (9) and +0.58 ± 0.43 (13).
Figure 1 shows the 9 trades of B in the first half, on the daily chart: two targets, six stops and a time exit. Figure 2 shows the 16 of the second half: nine targets and seven stops, 15 supply and a single demand, in a period that overall fell (with a rally in August). Many sit close together: on 7 September, the 8th and the 16th there are pairs of trades from the same move, so the 16 trades are worth fewer than 16 independent cases.
Read the chart in words
D1 gold chart from 18 December 2025 to 24 May 2026. Nine labels mark the trades at their exit: two at +2R, six at −1R, one a time exit.
| When | What | Price |
|---|---|---|
| 18 Feb 2026 | +2R | 4,860.70 |
| 27 Mar 2026 | −1R | 4,556.63 |
| 30 Mar 2026 | −1R | 4,582.63 |
| 7 Apr 2026 | −1R | 4,833.73 |
| 6 Apr 2026 | time exit | 4,703.87 |
| 6 Apr 2026 | −1R | 4,703.87 |
| 19 Apr 2026 | −1R | 4,744.78 |
| 24 Apr 2026 | −1R | 4,661.01 |
| 15 May 2026 | +2R | 4,661.13 |
Times in UTC.
Read the chart in words
D1 gold chart from 25 May to 5 October 2026. Sixteen labels mark the trades at their exit: nine at +2R and seven at −1R.
| When | What | Price |
|---|---|---|
| 22 Jun 2026 | −1R | 4,222.29 |
| 1 Jul 2026 | −1R | 4,116.82 |
| 30 Jun 2026 | +2R | 4,066.20 |
| 7 Jul 2026 | +2R | 4,182.91 |
| 7 Jul 2026 | +2R | 4,182.91 |
| 14 Jul 2026 | +2R | 4,106.24 |
| 10 Sept 2026 | −1R | 4,315.55 |
| 8 Sept 2026 | −1R | 4,446.54 |
| 7 Sept 2026 | −1R | 4,432.49 |
| 10 Sept 2026 | +2R | 4,438.79 |
| 8 Sept 2026 | +2R | 4,446.54 |
| 14 Sept 2026 | +2R | 4,359.96 |
| 14 Sept 2026 | +2R | 4,359.96 |
| 17 Sept 2026 | −1R | 4,387.02 |
| 17 Sept 2026 | −1R | 4,387.02 |
| 23 Sept 2026 | +2R | 4,372.54 |
Times in UTC.
Test 4: different thresholds
The swing thresholds (1.5 ATR on the daily chart for the trend, 1.5 ATR on H4 for the half of the range) are choices. Changing them, version B (2R, margin 0.25) gives, with the number of trades in brackets:
| D1 swing / H4 swing | 1 ATR | 1.5 ATR | 2 ATR | 3 ATR |
|---|---|---|---|---|
| 1 ATR | +0.42 (25) | +0.55 (23) | +0.28 (30) | +0.40 (36) |
| 1.5 ATR | +0.25 (27) | +0.35 (25) | +0.14 (32) | +0.27 (38) |
| 2 ATR | +0.15 (18) | +0.32 (18) | +0.21 (22) | +0.38 (28) |
| 3 ATR | −0.36 (20) | −0.36 (20) | −0.25 (21) | +0.18 (26) |
The result stays positive in 13 combinations out of 16, with values from +0.14 to +0.55, and turns negative (−0.25 and −0.36) only with the D1 swing at 3 ATR, which leaves very few swings. The errors are 0.25-0.36 R in every cell.
What can be said
- The filters together showed in this sample a signal beyond chance (1-3% of random trials).
- The signal survives different swing thresholds, but comes from 18-38 trades per cell, with errors of 0.25-0.36 R.
- It does not clearly survive the walk-forward: choosing in one half and testing in the other give +0.95 one way and 0.00 the other.
- It lives mostly in the second half of the period, with supply, in a period that overall fell. It is not shown to hold in a rising period (demand has 4 trades).
- It is the version chosen after looking at the data: it must be retested on new data.
A four-step routine
- Write down how many variants you tried, including the discarded ones.
- Compare the best with what chance finds with the same grid.
- Split the period in two and check that the result shows up in both halves.
- Change the filter thresholds: if the result vanishes with small changes, it was not robust.
Exercises
Chapter exercises
1. On the chart the letters are four trades of version B in the first half of the period. Which letter is the trade that reached the target?
Read the chart in words
| When | What | Price |
|---|---|---|
| 18 Feb 2026 | Point A | 4,860.70 |
| 27 Mar 2026 | Point B | 4,556.63 |
| 30 Mar 2026 | Point C | 4,582.63 |
| 7 Apr 2026 | Point D | 4,833.73 |
Times in UTC.
2. Why must the best variant of a grid of 96 be read together with the grid?
3. The mean of all the variants' mean R is +0.09, against a random median of −0.09 (95th percentile +0.05). What do you conclude?
4. Choice in the first half: +0.95 in the second. Choice in the second: 0.00 in the first. What do you conclude?
Have a question or want to share your exercise?
Post in the community, or join the free signals room on Telegram.
Educational content, not financial advice. Trading involves risk.