Part 4: Testing honestly · Chapter 14 of 14 · 17 min

A backtest told twice, mistakes, reference, glossary and final test

The same rule told in two ways: "+58 R, drawdown 6 R, profit factor 2.4" and "the best of 480, +0.74 ± 0.43 R, chance +0.28, out of sample +0.43 ± 0.73 over 25 trades, walk-forward +0.16 ± 0.33, protocol test +0.12 ± 0.73: interesting and unproven". Then the most frequent mistakes in chapter order, the quick reference with rules and measurements, the glossary and a final test of 15 questions.

A backtest told twice

Take the most beautiful rule of the grid (80-candle breakout, 1.5 ATR stop, 3 R target, duration 48).

The story told in a presentation. "Ten-month backtest on gold: +58.1 R over 79 trades, maximum drawdown 6.1 R, profit factor 2.37, positive in all five periods, and the neighbours are all positive." All true, and without a word to judge it.

The honest story. The same rule, with all the numbers of the book:

QuestionAnswer (chapter)
How many rules were tried?480: it is the best (2)
What does chance give for 480 trials?On random series the best is worth +0.28 R (maximum +0.53); with 10-day blocks +0.53 (maximum +0.80) (3, 9)
How much is left, removing the selection?About +0.45 R, with a lower limit of +0.02 (9)
How many degrees of freedom?Five parameters on 55 trades (11 per parameter) (4)
What does it do out of sample?+0.82 → +0.43 ± 0.73 over 25 trades (5)
And with monthly re-optimization?Walk-forward +0.16 ± 0.33 over 96 trades; with other windows from −0.13 to +0.45 (6)
Are there timing errors?No, with entry at the close and the delay test; with the mistake the same idea would give +0.83 (7)
Execution?Wide stops: no ambiguous candles; cost from 0 to 2 $ leaves +0.75 → +0.68 (8)
Does it beat chance?P-value of the single rule under 1 in 300; against 10-day blocks 1 series in 150 exceeds it (9, 10)
Robust?Neighbours +0.28/+0.66 (58%), like the best of noise (0.52); the N = 80 row stands out (11)
Stable over time?Positive in every period but the first makes 48%; two months out of ten make +29.7 R out of +58.1 (12)
Single test of the protocol?Choice +0.53 ± 0.50; test +0.12 ± 0.73 over 23 trades: inconclusive (13)
Figure 1. Cumulative R: example rule of chapter 1 (bars, −4.6 R), best of 480 (pink, +58.1 R), rule chosen with the protocol (blue, +31.3 R) and walk-forward from March (dashed, +15.6 R); the grey band is the second part. The more honest the procedure, the lower the curve.Data: gold, XAUUSDT perpetual future (real candles) · daily (D1) · from 15 Dec 2025 to 5 Oct 2026.
Read the chart in words

Daily gold chart with, below, four cumulative R curves: the best of 480 very high, the protocol choice and the walk-forward lower, the example rule near zero, and a grey band over the last third.

WhenWhatPrice
24 Jun 2026 → 5 Oct 2026second part

Times in UTC.

The rule is neither "a system" nor "a fluke". It is an interesting and unproven result: in this sample long-range breakouts did better than chance makes plausible, but with few trades, a single instrument, a favourable regime (chapter 12) and a gain that halves out of sample. It is the kind of result that deserves a forward test with small risk and rules written beforehand, not a real account. The difference between the two stories is not in the data: it is in the number of questions answered.

The most common mistakes

MistakeChapter
Writing the rule after seeing the chart; reading the sum of R without trades and error1
Presenting the best as "the system"; counting five almost identical rules as five confirmations2
Saying "I only tried three rules" when they are the survivors; comparing the best with zero instead of with the maximum of chance3
Adding a filter "because it improves the backtest"; saying the rule has one parameter after choosing among ten4
Tweaking the rule until it holds out of sample; splitting at random instead of in time5
Trying choice windows and reporting the best; comparing the walk-forward with the rule chosen looking at everything6
Trusting an exceptional result because "the code is correct"; entering at the open of a candle whose signal uses the close7
Choosing the assumption for ambiguous candles after seeing the result; testing a narrow stop on hourly candles8
Computing the p-value of the best as if it were the only one; choosing the most convenient null hypothesis; choosing on the sum9
Saying "p = 0.003, the rule works"; choosing the nicest shuffling10
Saying "it is robust, the neighbours are positive" without the comparison with noise11
Saying "it works in all periods" without removing the best period; believing ten months are "the market"12
Reading the test's +0.12 as "confirmed" or as "refuted"; tweaking after the test and retesting on the same data13

Quick reference: the rules

ConceptRule of the bookCh.
BacktestWritten rules a computer applies; declared data; result in R with cost; execution assumptions1, 8
Family of rulesFixed beforehand: 5 × 4 × 4 × 3 × 2 = 480; all results saved1, 2
TrialsCount them all, hidden ones included; best of N = maximum of chance over N3
ParametersCount the values tried; under 20 trades per free parameter the rule is fragile4
SplitIn time, decided beforehand: 65% choice, 35% test; the test is looked at once5
Walk-forwardChoose on previous data, trade the next period; the window is a parameter6
TimingEnter at the close of the signal or at the next open; delay test7
ExecutionStop first in ambiguous candles, non-minimal cost, whatever is missing goes in the worst direction8
P-valueOn the maximum of chance, not on the single rule; mean R, not sum; more than one null hypothesis9, 10
RobustnessOne-step neighbours, map, neighbours/best ratio compared with noise11
Periods4-6 periods, remove the best, regimes12
TestDecision rule written beforehand; "inconclusive" is a result; trades needed 7.85 σ²/μ²13

Quick reference: the measurements

Ch.Result
14,865 H1 candles (15 Dec 2025 - 6 Oct 2026); example rule 307 trades at −0.02 ± 0.11 R; grid of 480 rules, 108,309 trades
2Grid −0.017 R, 189 positive rules, 32 with lower limit > 0; best +0.74 ± 0.43 (79 tr., +58.1 R, factor 2.37, drawdown 6.1 R)
3On random series best of 1/5/20/480 rules: −0.03/+0.09/+0.16/+0.28; of 480: maximum +0.53, > +0.3 in a third of series
41 parameter: +0.05 → +0.01 (random); 5 parameters: +0.35 → +0.02; real data 3-5 parameters: +0.75-0.82 → +0.43-0.46
5Choice +0.82 ± 0.52 (55 tr.) → test +0.43 ± 0.73 (25); first decile +0.38 → +0.10; rank correlation +0.43 (random +0.17)
63-month walk-forward +0.16 ± 0.33 (96 tr.); random +0.01; windows 1/2/3/6 months: −0.13/+0.06/+0.16/+0.45
7Timing error: −0.02 → +0.83 R (92% wins); daily rule −0.15 → +1.61 (98%)
8Ambiguous candles: 189 → 273 positive rules; narrow stop −0.17 → +0.12; costs 0-2 $: mean −0.001 → −0.110, best +0.75 → +0.68
9Best on random (blocks of 1/4/10 days): +0.28/+0.43/+0.53; 65 rules with p ≤ 0.05 (24 expected), 59 breakouts
10P-values of the best: 0.003 (direction), 0 (timing), 0 (series); median and example 0.36-0.89
11Neighbours +0.28/+0.66 (mean +0.43); ratio 0.58 against 0.52; choice by neighbours: +0.12 in the test
12Fade negative in 5 periods out of 5, breakout positive in 4; rank correlation between periods +0.72/+0.29/−0.01/+0.03; the first period makes 48%
13Choice +0.53 ± 0.50 → test +0.12 ± 0.73 (23 tr.): inconclusive; on random the test is positive in 53% and ≥ +0.12 in 30%

All the measurements come from ten months of a single instrument (December 2025 - October 2026, gold perpetual future on Binance, prices only), from a grid of 480 breakout rules and from random series built by shuffling the same candles. None demonstrates an operational edge.

Glossary

The terms of the book are in the site glossary, where each has an example with numbers.

TermDefinition
Rule gridThe set of all combinations of the values chosen for the parameters of a rule, tried one by one on the same data.
Multiple testing and selectionThe fact that, by trying many rules on the same data and keeping the best, the result of the best overstates what awaits the rule on new data.
Researcher degrees of freedomAll the choices made by looking at the results: parameter values tried and discarded, timeframe, instrument, period, filters, versions of the rule.
Trades per parameterThe number of trades of a rule divided by the number of parameters the data chose: a measure of how much the result can be fitting.
Regression to the meanThe fact that rules chosen because they have the highest result on a sample tend, on new data, to a result closer to the average.
Single-use test partThe part of the data set aside to test the rule once, without using it to choose or to correct.
Look-ahead biasA timing error in which a backtest uses, at a point, information that did not yet exist at that moment.
Ambiguous candleA candle that touches both the stop and the target of a trade: with candles alone you do not know which came first.
Selection-adjusted p-valueThe probability that, among N rules with no edge, the best does equally well or better than the observed result: it is computed on the maximum of chance, not on a single rule.
Random comparison seriesA series built by shuffling the real candles (in blocks) and removing the trend, with the same volatility and without the link you want to measure.
Null hypothesis of the testWhat is broken by the shuffling: direction, entry timing or structure of the series. It decides which question the p-value answers.
Neighbour meanThe mean of the results of the rules that differ from the chosen one by a single parameter and by a single step: a measure of robustness.
Market regimeA stretch of market in which prices behave in a certain way (trend, range, high or low volatility): a rule may work in one regime and not in another.
Testing protocolThe set of rules written before looking at the results: family of rules, data split, execution assumptions, selection criterion, comparison with chance, decision rule and single test.

What to do now

  1. Write the process before looking at the results (chapter 13): family of rules, split, prudent assumptions, selection criterion, decision rule for the test.
  2. Keep the log of trials and always report "best of N".
  3. Compare every result with chance (shuffled series, more than one null hypothesis) and with the out-of-sample result.
  4. Check timing first: no datum used before it exists; delay test.
  5. If the verdict is inconclusive, proceed with small risk (book 11) and a forward test decided beforehand.
  6. For the basics: the book "Market structure", the book "Order blocks from A to Z", the book "FVG and liquidity", the book "SMC: the whole system", the book "Indicators from A to Z", the book "Elliott and Fibonacci", the book "Chart patterns", the book "Candlesticks", the book "Hours, sessions and news", the book "Support, resistance and levels" and the book "Risk and position sizing". For risk and discipline: the risk and psychology course.

Final test

Fifteen questions on the book. You pass the test with at least 12 correct answers.

Final test of the book

1. On the chart the lines are the entry (5,405.79 $), the stop (5,374.51 $) and the target (5,452.71 $) of a long trade of 2 March 2026 (UTC hours); the line at the upper left is the 30-candle high (5,388.00 $). Which letter is the candle in which the stop is hit?

Chart exercise. Data: gold, XAUUSDT perpetual future (real candles) · 1 hour (H1) · from 1 Mar 2026 17:00 UTC to 2 Mar 2026 21:00 UTC.
Read the chart in words
WhenWhatPrice
2 Mar 2026 07:00 UTCentry5,405.79
2 Mar 2026 07:00 UTCstop: −1 R (31.3 $)5,374.51
2 Mar 2026 07:00 UTCtarget: +1.5 R5,452.71
1 Mar 2026 23:00 UTC30-candle high5,388.00
2 Mar 2026 03:00 UTCPoint A5,367.99
2 Mar 2026 07:00 UTCPoint B5,420.00
2 Mar 2026 10:00 UTCPoint C5,399.46
2 Mar 2026 11:00 UTCPoint D5,397.81

Times in UTC.

2. On the chart the lines are the entry (4,638.15 $), the stop (4,609.62 $) and the target (4,680.95 $) of a long trade of 31 March 2026 (UTC hours), with the 30-candle high at 4,631.98 $. Which letter is the signal and entry candle?

Chart exercise. Data: gold, XAUUSDT perpetual future (real candles) · 1 hour (H1) · from 31 Mar 2026 02:00 UTC to 1 Apr 2026 05:00 UTC.
Read the chart in words
WhenWhatPrice
31 Mar 2026 16:00 UTCentry4,638.15
31 Mar 2026 16:00 UTCstop: −1 R (28.5 $)4,609.62
31 Mar 2026 16:00 UTCtarget: +1.5 R4,680.95
31 Mar 2026 08:00 UTC30-candle high4,631.98
31 Mar 2026 12:00 UTCPoint A4,597.99
31 Mar 2026 16:00 UTCPoint B4,651.33
31 Mar 2026 18:00 UTCPoint C4,670.00
31 Mar 2026 19:00 UTCPoint D4,683.60

Times in UTC.

3. On random series with no edge, how much is the best of 480 rules tried worth on average?

4. With 5 free parameters, on random series the best of the first part is worth +0.35 R. How much is it worth, roughly, in the second?

5. The first decile of the rules makes +0.38 R in the first part and +0.10 R in the second. What is the phenomenon called?

6. Why is every trade of a walk-forward out of sample?

7. You delay every entry by one candle and mean R goes from +0.83 to −0.02. What does it mean?

8. A narrow-stop rule makes −0.17 R with the prudent rule for ambiguous candles and +0.12 with the optimistic one. What do you do?

9. The best of 480 is worth +0.74 R. What must it be compared with, for the p-value?

10. With shuffled blocks of 10 days the best of 480 is worth on average +0.53 R on the random series, against +0.28 with one-day blocks. What does it mean?

11. The neighbours/best ratio is 0.58 for the real rule and 0.52 for the best of the random series. What does it say?

12. The best rule has 48% of its gain in the first of five periods, when gold rises 18%. What do you do?

13. The single test makes +0.12 ± 0.73 R over 23 trades. With the decision rule written beforehand (lower limit above zero = passed), what is the verdict?

14. The rank correlation between the results of the rules at cost 0 and at cost 1 $ is 0.976. What does it say?

15. You try 12 indicators on 3 timeframes with 4 stops each. How many trials did you run?

Have a question or want to share your exercise?

Post in the community, or join the free signals room on Telegram.

Educational content, not financial advice. Trading involves risk.