Part 4: The system under test · Chapter 12 of 14 · 9 min

Robustness and overfitting: how well a result holds

Four robustness tests on the system (a grid of 96 variants, a comparison with randomly assigned filters, two halves of the period, different thresholds): the filters together beat chance in 1-3% of random trials, but the result comes from 25 trades, nearly all supply in the second half of the period.

The problem

A result found after trying many variants is higher than one chosen in advance. If you try a hundred variants of a system with no edge, the best has a positive result. Chapter 11 gave two positive versions (B and C) put together after seeing the data. Here we measure how well the result holds with four tests.

Test 1: the grid

The variants are 96 (16 combinations of the four filters times six orders: 1.5R, 2R or 3R target, stop margin of 0.25 or 0.5 ATR). With at least 20 trades, 54 remain.

GroupVariants
With positive mean R28 out of 54
With positive mean R beyond one error9
With positive mean R beyond two errors1

The top five: trend plus half at 3R with a 0.5 ATR margin (25 trades, +0.82 ± 0.39 R), trend plus half at 3R with a 0.25 margin (+0.79 ± 0.40), trend plus take at 3R (23 trades, +0.74 ± 0.39 and +0.55 ± 0.40), trend plus half at 2R with a 0.5 margin (+0.41 ± 0.29). The bottom five run from −0.14 to −0.15 R (no filter and 1.5R, half only or break only, trend plus break). The mean of the variants' mean R is +0.09, the median 0.00.

Test 2: what chance finds

The filters are assigned to POI at random (shuffled across the 291) and the grid is repeated, 1,000 times. The best variant with at least 20 trades has, in this null case, a median mean R of +0.31 (5th percentile +0.06, 95th +0.70). The observed value (+0.82) is exceeded by 3% of the random trials. The mean of the mean R of all the variants is +0.09 against a random median of −0.09 (95th percentile +0.05): the observed mean is exceeded by 1.4% of the random trials.

With randomly assigned filters, then, the best variant of such a wide grid already has a positive R (+0.31 as the median). What stands out is not the highest result, but the fact that the set of variants sits above chance: in this sample the four filters together show a signal beyond chance in 1-3% of trials.

Test 3: two halves of the period

The walk-forward picks the best variant in one half and tests it in the other:

ChoiceVariant chosenIn the choice halfIn the other half
First halfTrend plus take, 3R, margin 0.510 trades, +0.47 ± 0.5813 trades, +0.95 ± 0.54
Second halfTrend plus half, 3R, margin 0.2516 trades, +1.23 ± 0.519 trades, 0.00 ± 0.58

The result is opposite in the two directions, with 9-16 trades and errors of 0.5. Version B in particular: first half (up to 24 May) 9 trades, −0.22 ± 0.44; second half 16 trades, +0.67 ± 0.38. By side: demand 4 trades (−0.26 ± 0.75), supply 21 (+0.46 ± 0.33). Version E: −0.21 ± 0.44 (9) and +0.58 ± 0.43 (13).

Figure 1 shows the 9 trades of B in the first half, on the daily chart: two targets, six stops and a time exit. Figure 2 shows the 16 of the second half: nine targets and seven stops, 15 supply and a single demand, in a period that overall fell (with a rally in August). Many sit close together: on 7 September, the 8th and the 16th there are pairs of trades from the same move, so the 16 trades are worth fewer than 16 independent cases.

Figure 1. The 9 trades of version B in the first half of the period (December-May): two targets, six stops, one time exit.Data: gold, XAUUSDT perpetual future (real candles) · daily (D1) · from 18 Dec 2025 to 24 May 2026.
Read the chart in words

D1 gold chart from 18 December 2025 to 24 May 2026. Nine labels mark the trades at their exit: two at +2R, six at −1R, one a time exit.

WhenWhatPrice
18 Feb 2026+2R4,860.70
27 Mar 2026−1R4,556.63
30 Mar 2026−1R4,582.63
7 Apr 2026−1R4,833.73
6 Apr 2026time exit4,703.87
6 Apr 2026−1R4,703.87
19 Apr 2026−1R4,744.78
24 Apr 2026−1R4,661.01
15 May 2026+2R4,661.13

Times in UTC.

Figure 2. The 16 trades of version B in the second half of the period (May-October): nine targets, seven stops, 15 supply and one demand.Data: gold, XAUUSDT perpetual future (real candles) · daily (D1) · from 25 May 2026 to 5 Oct 2026.
Read the chart in words

D1 gold chart from 25 May to 5 October 2026. Sixteen labels mark the trades at their exit: nine at +2R and seven at −1R.

WhenWhatPrice
22 Jun 2026−1R4,222.29
1 Jul 2026−1R4,116.82
30 Jun 2026+2R4,066.20
7 Jul 2026+2R4,182.91
7 Jul 2026+2R4,182.91
14 Jul 2026+2R4,106.24
10 Sept 2026−1R4,315.55
8 Sept 2026−1R4,446.54
7 Sept 2026−1R4,432.49
10 Sept 2026+2R4,438.79
8 Sept 2026+2R4,446.54
14 Sept 2026+2R4,359.96
14 Sept 2026+2R4,359.96
17 Sept 2026−1R4,387.02
17 Sept 2026−1R4,387.02
23 Sept 2026+2R4,372.54

Times in UTC.

Test 4: different thresholds

The swing thresholds (1.5 ATR on the daily chart for the trend, 1.5 ATR on H4 for the half of the range) are choices. Changing them, version B (2R, margin 0.25) gives, with the number of trades in brackets:

D1 swing / H4 swing1 ATR1.5 ATR2 ATR3 ATR
1 ATR+0.42 (25)+0.55 (23)+0.28 (30)+0.40 (36)
1.5 ATR+0.25 (27)+0.35 (25)+0.14 (32)+0.27 (38)
2 ATR+0.15 (18)+0.32 (18)+0.21 (22)+0.38 (28)
3 ATR−0.36 (20)−0.36 (20)−0.25 (21)+0.18 (26)

The result stays positive in 13 combinations out of 16, with values from +0.14 to +0.55, and turns negative (−0.25 and −0.36) only with the D1 swing at 3 ATR, which leaves very few swings. The errors are 0.25-0.36 R in every cell.

What can be said

  • The filters together showed in this sample a signal beyond chance (1-3% of random trials).
  • The signal survives different swing thresholds, but comes from 18-38 trades per cell, with errors of 0.25-0.36 R.
  • It does not clearly survive the walk-forward: choosing in one half and testing in the other give +0.95 one way and 0.00 the other.
  • It lives mostly in the second half of the period, with supply, in a period that overall fell. It is not shown to hold in a rising period (demand has 4 trades).
  • It is the version chosen after looking at the data: it must be retested on new data.

A four-step routine

  1. Write down how many variants you tried, including the discarded ones.
  2. Compare the best with what chance finds with the same grid.
  3. Split the period in two and check that the result shows up in both halves.
  4. Change the filter thresholds: if the result vanishes with small changes, it was not robust.

Exercises

Chapter exercises

1. On the chart the letters are four trades of version B in the first half of the period. Which letter is the trade that reached the target?

Chart exercise. Data: gold, XAUUSDT perpetual future (real candles) · daily (D1) · from 18 Dec 2025 to 24 May 2026.
Read the chart in words
WhenWhatPrice
18 Feb 2026Point A4,860.70
27 Mar 2026Point B4,556.63
30 Mar 2026Point C4,582.63
7 Apr 2026Point D4,833.73

Times in UTC.

2. Why must the best variant of a grid of 96 be read together with the grid?

3. The mean of all the variants' mean R is +0.09, against a random median of −0.09 (95th percentile +0.05). What do you conclude?

4. Choice in the first half: +0.95 in the second. Choice in the second: 0.00 in the first. What do you conclude?

Have a question or want to share your exercise?

Post in the community, or join the free signals room on Telegram.

Educational content, not financial advice. Trading involves risk.