Part 1: What a backtest is · Chapter 3 of 14 · 8 min

How many tests you ran

The more rules you try, the better the best one looks, even if the data are noise. On 300 random series (the same candles shuffled in day blocks, with no trend) the best of a single rule is worth −0.03 R, the best of 20 is worth +0.16, the best of 480 is worth on average +0.28 R (95th percentile +0.43). On the real data the best is +0.74: above all 300 random series, but the honest comparison starts from +0.28 and not from zero.

The missing number

If you flip a coin once and it comes up heads, it is not a fact. If you flip it 480 times and take the best flip, neither. A backtest works the same way: every rule tried is a flip, and the result you tell is the best. Its value depends on how many flips you made, a number that almost no backtest reports.

Let us measure it. To know what chance produces, we need a series with no edge and the same characteristics as the real data. We build it like this: take the real hourly candles, cut them into blocks of 24 (a day), shuffle the blocks at random and remove the average trend. The result is a series with the same volatility and the same candle shapes, but with no structure lasting more than a day. We make 300 of them.

The best of N rules

On these random series we draw N rules at random from the 480, try them, and keep the best (at least 30 trades). We repeat many times. The mean R of the best, on average (and at the 95th percentile), against the same calculation on the real data:

Rules triedRandom series: mean (95th percentile)Real data: mean
1−0.03 (+0.15)−0.01
5+0.09 (+0.23)+0.18
20+0.16 (+0.30)+0.33
50+0.20 (+0.35)+0.43
100+0.23 (+0.38)+0.51
240+0.26 (+0.41)+0.61
480+0.28 (+0.43)+0.68

On a series with no edge at all the best of five rules is already worth +0.09 R and the best of 480 is worth +0.28 R. That +0.28 is a gain that chance hands to whoever searches enough. On the real data the curve climbs higher (from +0.18 to +0.68): the data have an extra structure (chapter 12), but the mechanism is the same, and you cannot know how much of the climb is chance without the comparison.

The best of 480, on 300 random series

On each of the 300 random series we run all 480 rules and keep the best (at least 30 trades):

MeasureValue
Mean R of the best: average+0.284
5th percentile / median / 95th percentile+0.17 / +0.27 / +0.42
Maximum observed+0.53
Series where the best exceeds +0.3 R34%
Series where the best exceeds +0.5 R0%
Trades of the best (median, 5th-95th)129 (from 82 to 231)
Best on the real data+0.74 R, 79 trades

In one series out of three, pure chance hands out a rule above +0.3 R. The real best (+0.74) sits above all 300: it is a good sign, and we return to it in chapter 9. But it is read correctly like this: not "+0.74 against zero", but +0.74 against a +0.28 that chance gives for free to whoever tries 480 rules.

Figure 1. The best of the 480 rules on three random series (grey, bars and dashed; mean R +0.28, +0.37 and +0.48, final sums +30, +61 and +60 R) and the real best (blue, +58.1 R). The curves of the random series, though with no edge at all, look like the real one.Data: gold, XAUUSDT perpetual future (real candles) · daily (D1) · from 15 Dec 2025 to 5 Oct 2026.
Read the chart in words

Daily gold chart with, below, three cumulative R curves obtained on random series and one real curve, all rising.

One of the three random curves ends at +59.6 R, almost like the real one (+58.1). It does not mean the real one is random: it means that a curve this nice is not enough, alone, to say it is not.

480 are not 480 trials

If the rules were independent, 480 rules would give about 12 results with a lower limit above zero by chance (2.5% of 480). On the random series we find on average 4.6, with enormous variability: in some series zero, in others up to 45, and in 5% of series at least 20. On the real data there are 32. The rules of the grid resemble one another (they change one parameter at a time) and share the same candles, so they win and lose together: either many win, or few. That is why the number of rules with a good result says less than it seems to, and in a single series can be low or high by pure luck.

The same holds for rules in profit: on the random series they are on average 179 out of 480 (37%), on the real data 189. That 39% of a grid is positive is, in itself, normal.

Hidden trials

In the grid trials are counted: 480. In real life you decide the number, and it is almost always larger than the one you declare. An honest search counts:

What you changedTrials
Indicator parameters (lengths, thresholds)for each value tried
Stop, target and durationfor each combination
Timeframe (H1, H4, D1)for each timeframe
Instrumentsfor each instrument
Test periodfor each window looked at
Filters (hour of day, day of week, volatility)for each filter kept or discarded
Versions of the rulefor each version after seeing a result

Twelve indicators on three timeframes with four stops are 144 trials; add three filters and they are 1,296. Whoever presents "the rule" has often looked at hundreds of versions and remembers one. The remedy is simple and inconvenient: keep a log of everything you try, discarded rules too.

What to do with it

  1. Count the trials: write in the log every rule, parameter, timeframe and period tried.
  2. Report the number next to the result: "best of 480", and in the comparison start from the +0.28 that chance gives for that number of trials.
  3. Do not count relatives as independent trials, but do not remove them from the log: every variant looked at enters the count.
  4. Before searching, fix the family of rules: fewer trials, more value to the result.
  5. Set a limit: for example at most 50 variants, written beforehand.

Procedure in four steps

  1. Open a log and note every rule, parameter, timeframe, instrument and period you try.
  2. Fix, if you can, the family of rules before looking at the results.
  3. At the end of the search, write the total number of trials and how many were relatives.
  4. Compare the best result with what is obtained on the same family with random series.

Exercises

Chapter exercises

1. On random series the best of 5 rules is worth on average +0.09 R. How much is, on average, the best of 480?

2. You try 12 indicators on 3 timeframes with 4 stops each. How many trials did you run?

3. On 300 random series, in what share of series does the best of the 480 rules exceed +0.3 R?

4. Why are 480 rules not 480 independent trials?

Have a question or want to share your exercise?

Post in the community, or join the free signals room on Telegram.

Educational content, not financial advice. Trading involves risk.