Part 4: Testing honestly · Chapter 13 of 14 · 10 min
An honest testing process
We put the tools together in a process written beforehand: family of rules fixed, data split, prudent execution assumptions, a selection criterion with p-value and neighbours, a single out-of-sample test. On our 480 rules the process chooses a rule (80-candle breakout, 2 ATR stop, 3 R target, duration 48: +0.53 ± 0.50 R, p-value 0.003) and in the single test it makes +0.12 ± 0.73 R over 23 trades: inconclusive. On random series the same procedure gives a positive test in 53% of cases.
The process, in ten lines
An honest backtest is not a technique: it is a discipline decided before looking at the results. The process we use, with the chapters that motivate it:
- Hypothesis and family of rules, written beforehand: which idea, which parameters and which values (chapters 1 and 3).
- Data and split in time: choice part and test part, with the date decided beforehand (chapter 5).
- Prudent execution assumptions: ambiguous against, realistic entry, a cost that is not minimal (chapter 8).
- Timing check: no datum used before it exists, with the delay test (chapter 7).
- Log of trials: every rule and every parameter looked at (chapter 3).
- Selection criterion, written beforehand, on the choice part only: for example lower limit above zero and highest neighbour mean (chapter 11).
- Comparison with chance: p-value against random series and against the maximum (chapters 9 and 10).
- Decision rule for the test, written beforehand: what result counts as passed.
- Single test on the test part, once only, with trades and error (chapter 5).
- What you will do with the result: small risk, risk plan, forward test (book 11).
Applied to the grid
Step 1. Family: the 480 rules of chapter 1. Step 2. First part until 24 June 2026 (3,162 candles), second part from there (1,703). Steps 3-5. Prudent assumptions of chapter 8, entry at the close, log: 480 rules, no other variant looked at.
Step 6, choice, on the first part alone. Among the 480 rules, all with at least 30 trades, 46 have the lower limit of the interval above zero. Among these we take the one with the highest neighbour mean.
Step 7, comparison. In the first part, 82 rules out of 480 have a p-value ≤ 0.05 against the random series (about 24 would be expected by chance). The chosen rule has a p-value of 0.003.
The chosen rule is 80-candle breakout, 2 ATR stop, 3 R target, duration 48: 50 trades in the first part, +0.53 ± 0.50 R, neighbour mean +0.55. It is not the best by mean R (that one is +0.82 with a 1.5 ATR stop): the neighbours criterion prefers it because it sits in a less spiky zone.
Step 8, decision rule for the test (written beforehand). Suppose we wrote: "the test is passed if the lower limit of the interval is above zero; if the mean is positive but the interval contains zero, the result is inconclusive; if the mean is negative, the rule is discarded".
Step 9, single test, on the second part.
| Measure | Value |
|---|---|
| Trades | 23 |
| Mean R ± error | +0.12 ± 0.73 |
| Lower limit of the interval | −0.61 |
| Winning | 30% |
| Sum | +2.7 R |
| Deviation | 1.78 R |
The verdict, according to the rule written beforehand: inconclusive. The mean is positive (+0.12), but the interval (from −0.61 to +0.85) amply contains zero. The process has not shown anything: it has not refuted either.
Read the chart in words
Daily gold chart with, below, two cumulative R curves and a grey band over the last third of the period: the pink curve almost stops, the bar curve still rises a little.
| When | What | Price |
|---|---|---|
| 24 Jun 2026 → 5 Oct 2026 | single test |
Times in UTC.
For comparison: the naive choice (the highest mean R) gave +0.43 ± 0.73 over 25 trades in the same test (chapter 5). If we had chosen looking at the second part, the best would have been 80-candle breakout, 1 ATR stop, 3 R target, duration 48 (32 trades, +0.48 ± 0.68): it would look like a result in line with the others, but it would be a choice made on the test, and it would be worth nothing as a test. The mean of all 480 rules in the second part is −0.02.
Does the process work?
As for any method, we test it on chance. We apply the same process (choice of a rule with lower limit above zero and highest neighbour mean in the first part, single test in the second) to the 300 random series:
| Measure on random series | Value |
|---|---|
| Series with at least one candidate rule | 192 out of 300 |
| Mean R of the rule chosen in the first part | +0.34 |
| Mean R in the single test (5th - 95th percentile) | +0.01 (from −0.29 to +0.28) |
| Single test with positive mean R | 53% |
| Single test with mean R ≥ +0.12 (ours) | 30% |
The process does what it must do: on data with no edge the rule chosen, which in the first part looks good (+0.34 on average), in the single test is worth zero (+0.01). But it also shows that a single test with 23 trades is a weak test: in 53% of cases with no edge at all the test comes out positive, and in 30% it makes +0.12 or more, like ours. The +0.12 cannot be told apart from chance, with this number of trades.
What to do after an inconclusive verdict
An inconclusive result is the honest answer with few data. It has three practical consequences.
- Do not trade the rule as if it were proven. At most, with small risk (book 11: 0.5-1%) and a written plan, as a forward test.
- Compute how much data you need to decide. With a deviation of 1.78 R, to tell an edge of +0.4 R you need about 155 trades (power 80%), that is about 21 months at this rule's pace (7-8 per month); for +0.25 R you need about 400, more than four years. For +0.12 R, about 1,800. The forward test is slow because the rule trades little.
- Decide beforehand the decision rule of the forward test: number of trades (for example 150), threshold (lower limit above zero), what you will do if it passes and if not.
The forward test (data after 6 October 2026) is the test we lack. It has not been done here: it is work to do with rules and criterion written beforehand, and with the agreement of whoever decides.
The checklist
Before believing a backtest, ten questions, with the answers written down:
| Question | Answer in our case |
|---|---|
| How many rules were tried? | 480, logged |
| Was the family written beforehand? | Yes (fixed grid of chapter 1) |
| Test part never seen to choose? | Yes (from 24 June 2026), looked at once |
| Prudent execution? | Yes (stop first, cost 0.3 $, entry at the close) |
| Timing errors excluded? | Yes, with the delay test on the signal |
| Comparison with chance? | Yes: p-value 0.003; maximum of chance +0.28-0.53 |
| Robustness to neighbours? | Positive neighbours; the N = 80 row stands out |
| Stability across periods? | Positive in every period; 48% from the first |
| Out-of-sample result? | +0.12 ± 0.73 over 23 trades: inconclusive |
| What do you do? | Small risk, written plan, forward test |
What to do with it
- Write the process beforehand, with the decision rule for the test.
- Look at the test once and change nothing afterwards.
- Accept "inconclusive" when trades are few: it is not a failure.
- Compute how much data you need to decide and organise the forward test.
- Keep the log of everything, also of the rules that did not pass.
Procedure in four steps
- Write family, split, execution assumptions, selection criterion and decision rule before the results.
- Choose on the first part and compare with chance (p-value, maximum).
- Test once on the second and apply the decision rule written beforehand.
- If the verdict is inconclusive, compute the data needed and organise a forward test with small risk.
Exercises
Chapter exercises
1. The single test gives +0.12 ± 0.73 R over 23 trades. With the decision rule written beforehand (lower limit above zero = passed), what is the verdict?
2. Why is the decision rule for the test written before looking at the result?
3. On random series the process gives a positive single test in 53% of cases. What does it say?
4. You looked at the second part, changed a parameter and redid the test. What must you do?
Have a question or want to share your exercise?
Post in the community, or join the free signals room on Telegram.
Educational content, not financial advice. Trading involves risk.