Part 3: Chance and robustness · Chapter 10 of 14 · 8 min
The permutation test
A permutation test shuffles part of the backtest and asks how often the shuffling does equally well or better than the real data. On the best rule (+0.74 R) random direction gives on average +0.23 (p-value 0.003), random timing +0.04 (p-value 0) and the shuffled series +0.05 (p-value 0). The median rule (−0.05) and the example rule (−0.02) stay in the middle of chance (p-values from 0.36 to 0.89). But the p-value of a single rule does not account for the 480.
The idea
A permutation test answers a simple question: "if the link between signal and result did not exist, how often would I have seen a result like this?". To find out you break the link with a shuffling, repeat it many times and look at where the real result falls in the distribution of the shuffled results. The share of shufflings that do equally well or better is the p-value.
The strength of the method is that it assumes nothing about the shape of the data (no bell curves, no formulas): it uses the data themselves. Its weak point is that shuffling means deciding what is broken, and breaking different things answers different questions. We use three.
The three shufflings
- Random direction. You keep the same entry moments, but assign each trade a direction (buy or sell) at random. It breaks the link between signal and direction: it asks "is the direction chosen by the rule worth more than a random direction?".
- Random timing. You keep stop, target and duration, and choose at random when to enter (with the same frequency of signals), with a random direction. It breaks the link between signal and moment: it asks "is entering at the moment of the breakout worth more than entering at random, with the same exits?".
- Shuffled series. You apply the same rule to the 300 random series of chapter 3 (candles shuffled in one-day blocks, with no trend). It breaks all the structure of the series lasting more than a day: it asks "this rule, on a series with no structure, does like this?".
On the best rule, on a median one and on the example one
Let us take three rules: the best of the grid, the median rule (by mean R) and the example rule of chapter 1. 1,000 trials for the first two shufflings, 300 series for the third.
| Rule | Real | Random direction: mean (95th perc.), p-value | Random timing: mean (95th perc.), p-value | Shuffled series: mean (95th perc.), p-value |
|---|---|---|---|---|
| Best: breakout 80, 1.5 ATR stop, 3 R target, duration 48 | +0.74 ± 0.43 (79 tr.) | +0.23 (+0.53), 0.003 | +0.04 (+0.28), 0 | +0.05 (+0.30), 0 |
| Median: fade 50, 2 ATR stop, 2 R target, duration 48 | −0.05 ± 0.23 (132 tr.) | +0.10 (+0.31), 0.89 | +0.02 (+0.21), 0.73 | −0.04 (+0.12), 0.50 |
| Example: breakout 20, 1 ATR stop, 1 R target, duration 24 | −0.02 ± 0.11 (307 tr.) | −0.03 (+0.06), 0.36 | −0.03 (+0.07), 0.42 | −0.00 (+0.06), 0.61 |
What they say.
- The median rule and the example rule sit in the middle of chance in all three tests (p-values from 0.36 to 0.89). No shuffling tells them apart. It is what a working test must do: give a p-value around 0.5 to a rule with no edge.
- The best sits outside chance in all three, but at different distances. Against random direction (p = 0.003) chance reaches +0.53 at the 95th percentile; against random timing +0.28; against the shuffled series +0.30.
- Random timing separates the moment from the exits. With a 1.5 ATR stop, a 3 R target and duration 48, entering at random the rule is worth +0.04: the gain does not come from the shape of the exits, but from entering on the breakout (in the ten months). This removes an objection: that "a big target always does well".
- Random direction gives +0.23 for the best, not zero. With the direction at random half of the trades keep the rule's direction, and the rule, by construction, earns in that direction. It is a prudent comparison: the reference value is not zero, and the +0.74 exceeds it all the same.
Read the chart in words
Daily gold chart with, below, the real curve rising strongly and two lower reference curves (median and 95th percentile of random direction).
What a p-value does not say
Three limits, not to be lost sight of.
A p-value of 0.003 is not "a 0.3% probability that the rule is wrong". It says: if there were no link, a shuffling would do equally well or better about one time in 333. It is a probability about the data, not about the rule.
The p-value of the single rule does not count the 480 trials. With 480 rules tried, even by chance alone you find several with a small p-value: chapter 9 corrects it by comparing with the maximum. On 60 rules taken one in eight from the grid (random direction, 200 trials each), the p-value is ≤ 0.05 for 14 (23%), whereas by chance it would be 5%: more than expected, and tilted towards breakouts.
The shuffling must break only what you want to break. The series shuffled in one-day blocks also breaks slow structures (chapter 9), and for that reason it is an easy comparison to beat for an 80-candle rule. Random timing and random direction, which leave the series intact, are stricter on that point.
How to do it well
- Write what you break (direction, timing, series) and what question you are asking.
- Use enough trials: 1,000 for a p-value around 0.01 (with fewer, the minimum p-value is 1 in N).
- Do more than one shuffling and report all the p-values.
- Do not use the p-value alone: read it with the interval, the effect, the out-of-sample tests and the correction for the number of rules.
- Check the test with a rule with no edge: it must give a p-value around 0.5.
Procedure in four steps
- Choose the shufflings (direction, timing, series) and write what each breaks.
- For each shuffling, at least 1,000 trials (300 series for the one on the series).
- Report mean, 95th percentile and p-value for each, and check them on a rule with no edge.
- Combine with the correction for the rules tried (chapter 9) and with the out-of-sample test.
Exercises
Chapter exercises
1. A rule with no edge (the median) has a p-value of 0.89 with random direction. How is it read?
2. What is the random-timing test, with the same exits, for?
3. With random direction the best rule gives on average +0.23 R and not zero. Why?
4. The p-value of the best rule is 0.003. Which sentence is right?
Have a question or want to share your exercise?
Post in the community, or join the free signals room on Telegram.
Educational content, not financial advice. Trading involves risk.