Part 4: Testing honestly · Chapter 14 of 14 · 17 min
A backtest told twice, mistakes, reference, glossary and final test
The same rule told in two ways: "+58 R, drawdown 6 R, profit factor 2.4" and "the best of 480, +0.74 ± 0.43 R, chance +0.28, out of sample +0.43 ± 0.73 over 25 trades, walk-forward +0.16 ± 0.33, protocol test +0.12 ± 0.73: interesting and unproven". Then the most frequent mistakes in chapter order, the quick reference with rules and measurements, the glossary and a final test of 15 questions.
A backtest told twice
Take the most beautiful rule of the grid (80-candle breakout, 1.5 ATR stop, 3 R target, duration 48).
The story told in a presentation. "Ten-month backtest on gold: +58.1 R over 79 trades, maximum drawdown 6.1 R, profit factor 2.37, positive in all five periods, and the neighbours are all positive." All true, and without a word to judge it.
The honest story. The same rule, with all the numbers of the book:
| Question | Answer (chapter) |
|---|---|
| How many rules were tried? | 480: it is the best (2) |
| What does chance give for 480 trials? | On random series the best is worth +0.28 R (maximum +0.53); with 10-day blocks +0.53 (maximum +0.80) (3, 9) |
| How much is left, removing the selection? | About +0.45 R, with a lower limit of +0.02 (9) |
| How many degrees of freedom? | Five parameters on 55 trades (11 per parameter) (4) |
| What does it do out of sample? | +0.82 → +0.43 ± 0.73 over 25 trades (5) |
| And with monthly re-optimization? | Walk-forward +0.16 ± 0.33 over 96 trades; with other windows from −0.13 to +0.45 (6) |
| Are there timing errors? | No, with entry at the close and the delay test; with the mistake the same idea would give +0.83 (7) |
| Execution? | Wide stops: no ambiguous candles; cost from 0 to 2 $ leaves +0.75 → +0.68 (8) |
| Does it beat chance? | P-value of the single rule under 1 in 300; against 10-day blocks 1 series in 150 exceeds it (9, 10) |
| Robust? | Neighbours +0.28/+0.66 (58%), like the best of noise (0.52); the N = 80 row stands out (11) |
| Stable over time? | Positive in every period but the first makes 48%; two months out of ten make +29.7 R out of +58.1 (12) |
| Single test of the protocol? | Choice +0.53 ± 0.50; test +0.12 ± 0.73 over 23 trades: inconclusive (13) |
Read the chart in words
Daily gold chart with, below, four cumulative R curves: the best of 480 very high, the protocol choice and the walk-forward lower, the example rule near zero, and a grey band over the last third.
| When | What | Price |
|---|---|---|
| 24 Jun 2026 → 5 Oct 2026 | second part |
Times in UTC.
The rule is neither "a system" nor "a fluke". It is an interesting and unproven result: in this sample long-range breakouts did better than chance makes plausible, but with few trades, a single instrument, a favourable regime (chapter 12) and a gain that halves out of sample. It is the kind of result that deserves a forward test with small risk and rules written beforehand, not a real account. The difference between the two stories is not in the data: it is in the number of questions answered.
The most common mistakes
| Mistake | Chapter |
|---|---|
| Writing the rule after seeing the chart; reading the sum of R without trades and error | 1 |
| Presenting the best as "the system"; counting five almost identical rules as five confirmations | 2 |
| Saying "I only tried three rules" when they are the survivors; comparing the best with zero instead of with the maximum of chance | 3 |
| Adding a filter "because it improves the backtest"; saying the rule has one parameter after choosing among ten | 4 |
| Tweaking the rule until it holds out of sample; splitting at random instead of in time | 5 |
| Trying choice windows and reporting the best; comparing the walk-forward with the rule chosen looking at everything | 6 |
| Trusting an exceptional result because "the code is correct"; entering at the open of a candle whose signal uses the close | 7 |
| Choosing the assumption for ambiguous candles after seeing the result; testing a narrow stop on hourly candles | 8 |
| Computing the p-value of the best as if it were the only one; choosing the most convenient null hypothesis; choosing on the sum | 9 |
| Saying "p = 0.003, the rule works"; choosing the nicest shuffling | 10 |
| Saying "it is robust, the neighbours are positive" without the comparison with noise | 11 |
| Saying "it works in all periods" without removing the best period; believing ten months are "the market" | 12 |
| Reading the test's +0.12 as "confirmed" or as "refuted"; tweaking after the test and retesting on the same data | 13 |
Quick reference: the rules
| Concept | Rule of the book | Ch. |
|---|---|---|
| Backtest | Written rules a computer applies; declared data; result in R with cost; execution assumptions | 1, 8 |
| Family of rules | Fixed beforehand: 5 × 4 × 4 × 3 × 2 = 480; all results saved | 1, 2 |
| Trials | Count them all, hidden ones included; best of N = maximum of chance over N | 3 |
| Parameters | Count the values tried; under 20 trades per free parameter the rule is fragile | 4 |
| Split | In time, decided beforehand: 65% choice, 35% test; the test is looked at once | 5 |
| Walk-forward | Choose on previous data, trade the next period; the window is a parameter | 6 |
| Timing | Enter at the close of the signal or at the next open; delay test | 7 |
| Execution | Stop first in ambiguous candles, non-minimal cost, whatever is missing goes in the worst direction | 8 |
| P-value | On the maximum of chance, not on the single rule; mean R, not sum; more than one null hypothesis | 9, 10 |
| Robustness | One-step neighbours, map, neighbours/best ratio compared with noise | 11 |
| Periods | 4-6 periods, remove the best, regimes | 12 |
| Test | Decision rule written beforehand; "inconclusive" is a result; trades needed 7.85 σ²/μ² | 13 |
Quick reference: the measurements
| Ch. | Result |
|---|---|
| 1 | 4,865 H1 candles (15 Dec 2025 - 6 Oct 2026); example rule 307 trades at −0.02 ± 0.11 R; grid of 480 rules, 108,309 trades |
| 2 | Grid −0.017 R, 189 positive rules, 32 with lower limit > 0; best +0.74 ± 0.43 (79 tr., +58.1 R, factor 2.37, drawdown 6.1 R) |
| 3 | On random series best of 1/5/20/480 rules: −0.03/+0.09/+0.16/+0.28; of 480: maximum +0.53, > +0.3 in a third of series |
| 4 | 1 parameter: +0.05 → +0.01 (random); 5 parameters: +0.35 → +0.02; real data 3-5 parameters: +0.75-0.82 → +0.43-0.46 |
| 5 | Choice +0.82 ± 0.52 (55 tr.) → test +0.43 ± 0.73 (25); first decile +0.38 → +0.10; rank correlation +0.43 (random +0.17) |
| 6 | 3-month walk-forward +0.16 ± 0.33 (96 tr.); random +0.01; windows 1/2/3/6 months: −0.13/+0.06/+0.16/+0.45 |
| 7 | Timing error: −0.02 → +0.83 R (92% wins); daily rule −0.15 → +1.61 (98%) |
| 8 | Ambiguous candles: 189 → 273 positive rules; narrow stop −0.17 → +0.12; costs 0-2 $: mean −0.001 → −0.110, best +0.75 → +0.68 |
| 9 | Best on random (blocks of 1/4/10 days): +0.28/+0.43/+0.53; 65 rules with p ≤ 0.05 (24 expected), 59 breakouts |
| 10 | P-values of the best: 0.003 (direction), 0 (timing), 0 (series); median and example 0.36-0.89 |
| 11 | Neighbours +0.28/+0.66 (mean +0.43); ratio 0.58 against 0.52; choice by neighbours: +0.12 in the test |
| 12 | Fade negative in 5 periods out of 5, breakout positive in 4; rank correlation between periods +0.72/+0.29/−0.01/+0.03; the first period makes 48% |
| 13 | Choice +0.53 ± 0.50 → test +0.12 ± 0.73 (23 tr.): inconclusive; on random the test is positive in 53% and ≥ +0.12 in 30% |
All the measurements come from ten months of a single instrument (December 2025 - October 2026, gold perpetual future on Binance, prices only), from a grid of 480 breakout rules and from random series built by shuffling the same candles. None demonstrates an operational edge.
Glossary
The terms of the book are in the site glossary, where each has an example with numbers.
| Term | Definition |
|---|---|
| Rule grid | The set of all combinations of the values chosen for the parameters of a rule, tried one by one on the same data. |
| Multiple testing and selection | The fact that, by trying many rules on the same data and keeping the best, the result of the best overstates what awaits the rule on new data. |
| Researcher degrees of freedom | All the choices made by looking at the results: parameter values tried and discarded, timeframe, instrument, period, filters, versions of the rule. |
| Trades per parameter | The number of trades of a rule divided by the number of parameters the data chose: a measure of how much the result can be fitting. |
| Regression to the mean | The fact that rules chosen because they have the highest result on a sample tend, on new data, to a result closer to the average. |
| Single-use test part | The part of the data set aside to test the rule once, without using it to choose or to correct. |
| Look-ahead bias | A timing error in which a backtest uses, at a point, information that did not yet exist at that moment. |
| Ambiguous candle | A candle that touches both the stop and the target of a trade: with candles alone you do not know which came first. |
| Selection-adjusted p-value | The probability that, among N rules with no edge, the best does equally well or better than the observed result: it is computed on the maximum of chance, not on a single rule. |
| Random comparison series | A series built by shuffling the real candles (in blocks) and removing the trend, with the same volatility and without the link you want to measure. |
| Null hypothesis of the test | What is broken by the shuffling: direction, entry timing or structure of the series. It decides which question the p-value answers. |
| Neighbour mean | The mean of the results of the rules that differ from the chosen one by a single parameter and by a single step: a measure of robustness. |
| Market regime | A stretch of market in which prices behave in a certain way (trend, range, high or low volatility): a rule may work in one regime and not in another. |
| Testing protocol | The set of rules written before looking at the results: family of rules, data split, execution assumptions, selection criterion, comparison with chance, decision rule and single test. |
What to do now
- Write the process before looking at the results (chapter 13): family of rules, split, prudent assumptions, selection criterion, decision rule for the test.
- Keep the log of trials and always report "best of N".
- Compare every result with chance (shuffled series, more than one null hypothesis) and with the out-of-sample result.
- Check timing first: no datum used before it exists; delay test.
- If the verdict is inconclusive, proceed with small risk (book 11) and a forward test decided beforehand.
- For the basics: the book "Market structure", the book "Order blocks from A to Z", the book "FVG and liquidity", the book "SMC: the whole system", the book "Indicators from A to Z", the book "Elliott and Fibonacci", the book "Chart patterns", the book "Candlesticks", the book "Hours, sessions and news", the book "Support, resistance and levels" and the book "Risk and position sizing". For risk and discipline: the risk and psychology course.
Final test
Fifteen questions on the book. You pass the test with at least 12 correct answers.
Final test of the book
1. On the chart the lines are the entry (5,405.79 $), the stop (5,374.51 $) and the target (5,452.71 $) of a long trade of 2 March 2026 (UTC hours); the line at the upper left is the 30-candle high (5,388.00 $). Which letter is the candle in which the stop is hit?
Read the chart in words
| When | What | Price |
|---|---|---|
| 2 Mar 2026 07:00 UTC | entry | 5,405.79 |
| 2 Mar 2026 07:00 UTC | stop: −1 R (31.3 $) | 5,374.51 |
| 2 Mar 2026 07:00 UTC | target: +1.5 R | 5,452.71 |
| 1 Mar 2026 23:00 UTC | 30-candle high | 5,388.00 |
| 2 Mar 2026 03:00 UTC | Point A | 5,367.99 |
| 2 Mar 2026 07:00 UTC | Point B | 5,420.00 |
| 2 Mar 2026 10:00 UTC | Point C | 5,399.46 |
| 2 Mar 2026 11:00 UTC | Point D | 5,397.81 |
Times in UTC.
2. On the chart the lines are the entry (4,638.15 $), the stop (4,609.62 $) and the target (4,680.95 $) of a long trade of 31 March 2026 (UTC hours), with the 30-candle high at 4,631.98 $. Which letter is the signal and entry candle?
Read the chart in words
| When | What | Price |
|---|---|---|
| 31 Mar 2026 16:00 UTC | entry | 4,638.15 |
| 31 Mar 2026 16:00 UTC | stop: −1 R (28.5 $) | 4,609.62 |
| 31 Mar 2026 16:00 UTC | target: +1.5 R | 4,680.95 |
| 31 Mar 2026 08:00 UTC | 30-candle high | 4,631.98 |
| 31 Mar 2026 12:00 UTC | Point A | 4,597.99 |
| 31 Mar 2026 16:00 UTC | Point B | 4,651.33 |
| 31 Mar 2026 18:00 UTC | Point C | 4,670.00 |
| 31 Mar 2026 19:00 UTC | Point D | 4,683.60 |
Times in UTC.
3. On random series with no edge, how much is the best of 480 rules tried worth on average?
4. With 5 free parameters, on random series the best of the first part is worth +0.35 R. How much is it worth, roughly, in the second?
5. The first decile of the rules makes +0.38 R in the first part and +0.10 R in the second. What is the phenomenon called?
6. Why is every trade of a walk-forward out of sample?
7. You delay every entry by one candle and mean R goes from +0.83 to −0.02. What does it mean?
8. A narrow-stop rule makes −0.17 R with the prudent rule for ambiguous candles and +0.12 with the optimistic one. What do you do?
9. The best of 480 is worth +0.74 R. What must it be compared with, for the p-value?
10. With shuffled blocks of 10 days the best of 480 is worth on average +0.53 R on the random series, against +0.28 with one-day blocks. What does it mean?
11. The neighbours/best ratio is 0.58 for the real rule and 0.52 for the best of the random series. What does it say?
12. The best rule has 48% of its gain in the first of five periods, when gold rises 18%. What do you do?
13. The single test makes +0.12 ± 0.73 R over 23 trades. With the decision rule written beforehand (lower limit above zero = passed), what is the verdict?
14. The rank correlation between the results of the rules at cost 0 and at cost 1 $ is 0.976. What does it say?
15. You try 12 indicators on 3 timeframes with 4 stops each. How many trials did you run?
Have a question or want to share your exercise?
Post in the community, or join the free signals room on Telegram.
Educational content, not financial advice. Trading involves risk.