Part 3: Chance and robustness · Chapter 9 of 14 · 9 min
The best of many attempts
The p-value of a rule chosen beforehand is read against chance; that of the best of 480 must be read against the best that chance produces over 480. On 300 random series the best is worth on average +0.28 R (maximum +0.53); ours, +0.74, exceeds them all. But with random series that keep 10-day blocks the best is worth on average +0.53 (maximum +0.80) and ours exceeds 149 out of 150: the answer depends on what we call "chance".
The p-value, of a single rule and of the best of many
The p-value is the probability of obtaining, in the absence of an edge, a result equal to or better than the one observed. It is computed by comparing the result with those obtained on random series.
If you had written the rule before looking at the data, the comparison is simple: your rule against the same rule on 300 random series. But if the rule is the best of 480, the right comparison is different: you must ask not "how often does chance give +0.74 with this rule?", but "how often does chance give +0.74 as the best of 480?". The two probabilities are very different.
For our best rule (80-candle breakout, 1.5 ATR stop, 3 R target, duration 48; +0.74 ± 0.43 over 79 trades):
| Comparison | Result |
|---|---|
| The same rule on 300 random series | mean +0.05, 95th percentile +0.30, maximum +0.51 |
| Series with the same rule ≥ +0.74 | 0 out of 300 (p-value of the single rule: under 1 in 300) |
| The best of the 480 on 300 random series | mean +0.284, maximum +0.53 |
| Series with the best ≥ +0.74 | 0 out of 300 (p-value after the choice: under 1 in 300) |
In this comparison the result beats chance in both ways. The value to start from to judge it is not zero, but +0.28: if we remove from the +0.74 the +0.28 that selection gives on average, +0.45 remains (and the lower limit of the interval, +0.31, becomes +0.02). The correction is rough, but it gives the idea: the true gain that remains, if there is one, is about half of what it seems, and its interval touches zero.
Read the chart in words
Daily gold chart with, below, three cumulative R curves: the median and the 95th percentile of the best on random series, and the real best, close to the 95th percentile.
Sum or mean: the measure matters
The comparison above uses the mean R per trade. If you use the sum of R the picture changes: the best on random series has a median final sum of +36.0 R (95th percentile +62.9, maximum +82.5), and 27 series out of 300 (9%) do equally well or better than our +58.1. The reason is that the random best has more trades (median 129 against 79) and the sum grows with the trades. Choosing on the sum rewards the rule that trades more, not the one that earns more per trade. With the mean, the real best is out of reach of chance; with the sum, it is not. Which to use is decided beforehand, and for the value of an edge the correct measure is the mean R with its error.
Rule by rule
A second way of looking at the problem: for each of the 480 rules, the p-value against the 300 random series (the same rule, on different series). If all were noise, we would expect 5% of p-values under 0.05 and 1% under 0.01.
| p-value of the rule | Rules observed | Rules expected by chance |
|---|---|---|
| ≤ 0.05 | 65 | 24 |
| ≤ 0.01 | 33 | 4.8 |
| 0 (no random series does equally well or better) | 16 | 0 |
They are many more than expected: 65 instead of 24. And they are not distributed at random: 59 are breakouts and 6 fades. In the real data breakouts do better than chance and fades do not (chapter 12). It is information about the sample: something in the ten months rewards the rules that follow the movement. It is not yet proof of an edge that lasts: it says the sample has a structure that the random series (shuffled in one-day blocks) do not have.
It depends on what you call "chance"
The random series of chapter 3 are shuffled in one-day blocks: they remove any structure lasting more than a day. But an 80-candle breakout rule (more than three days) lives precisely on movements that last days. By shuffling less, some of that structure remains. Let us redo the comparison with longer blocks:
| Shuffled blocks | Best of 480: mean (95th percentile, maximum) | Series with the best ≥ +0.74 | Series with the best ≥ +0.5 |
|---|---|---|---|
| 24 candles (one day), 300 series | +0.28 (+0.42, +0.53) | 0 out of 300 | 0% |
| 96 candles (about 4 days), 150 series | +0.43 (+0.61, +0.69) | 0 out of 150 | 23% |
| 240 candles (about 10 days), 150 series | +0.53 (+0.66, +0.80) | 1 out of 150 | 65% |
The longer the blocks, the more the chance "resembles" the real data, and the higher the maximum it produces. With 10-day blocks the best of 480 is worth on average +0.53 and in 65% of series exceeds +0.5. Ours, +0.74, stays on top (one series in 150 exceeds it), but the margin narrows: from "+0.74 against +0.28" to "+0.74 against +0.53".
What do the data say about structure? The autocorrelation of non-overlapping returns is +0.049 at 24 candles, +0.013 at 96 and −0.009 at 240 (error about ±0.28 at 96 candles): indistinguishable from zero at all scales. The random series have +0.007, −0.004 and −0.051. Returns, taken in blocks, do not repeat; what the shuffled series lose is something else (for example the fact that the large moves of those months are concentrated in a few stretches with the same direction). Chapter 10 explains how to choose the comparison.
One last check: is the in-sample best positive out of sample?
On random series, the best of the first part is positive in the second in 55% of cases: just above chance. If for our rule the answer (chapter 5) is "positive, +0.43 over 25 trades", it is one more datum, not a proof.
What to do with it
- Compare the best with the maximum that chance produces for the same number of trials, not with zero.
- Use the mean R, not the sum, to judge the quality of a rule.
- Count how many rules beat chance (p-value per rule) and compare with the expected (5%).
- Choose and declare the null hypothesis: what does the random series keep? Try more than one comparison.
- Read p-value, interval and out-of-sample together: none of the three is enough alone.
Procedure in four steps
- Build random series with a written, declared procedure (blocks, trend removed).
- Run all the rules on each and save the best.
- Compare the real best with the distribution of the maximum; also report the p-value per rule.
- Repeat with another null hypothesis (longer blocks) and report all the results.
Exercises
Chapter exercises
1. On random series the best of 480 rules has mean +0.28 and maximum +0.53; yours is +0.74. What do you compare, for the p-value?
2. 65 rules out of 480 have p-value ≤ 0.05 against the random series, where by chance there would be about 24; 59 are breakouts. How is it read?
3. With shuffled blocks of 10 days the best of 480 is worth on average +0.53 and in 65% of series exceeds +0.5. What does it mean?
4. Why can choosing the rule with the highest sum of R mislead?
Have a question or want to share your exercise?
Post in the community, or join the free signals room on Telegram.
Educational content, not financial advice. Trading involves risk.