FREE BOOK · BACKTEST

Backtest and overfitting from A to Z: how not to fool yourself with data

The Academy's twelfth book in 14 chapters: what a backtest is, the test that looks perfect, how many tests you ran, parameters and fitting, in-sample and out-of-sample, walk-forward, looking into the future, costs and execution, the best of many attempts, the permutation test, parameter robustness, periods and regimes and an honest testing process. Each chapter explains how it works, what we tried on 480 rules and on random series with real gold candles and what to do with it. With exercises, a glossary and a final test.

14 chapters · 4 modules · 2 hours of reading, approx. · updated 2026-11-20

How to use this book

Every chapter has figures on real gold candles, four exercises with the solution explained and a recap of the key points. Do the exercises before you move on to the next chapter. Educational content, not financial advice.

Part 1: What a backtest is

What a backtest is, the test that looks perfect, how many tests you ran and parameters.

  1. 1What a backtest isA backtest applies written rules to past data and records the result of each trade in R. In this book we use 4,865 hourly gold candles (15 December 2025 - 6 October 2026) and a family of 480 breakout rules. The example rule (20-candle high, 1 ATR stop, 1 R target) gives 307 trades at −0.02 ± 0.11 R: the backtest tells you what would have happened with those rules and that data, not what will happen. · 9 min
  2. 2The backtest that looks perfectTrying the 480 rules on the whole period, the best (80-candle breakout, 1.5 ATR stop, 3 R target, 48-candle duration) gives 79 trades at +0.74 R, +58.1 R in total, profit factor 2.37 and a drawdown of only 6.1 R. It looks like a system. But it is the best of 480, the mean of all the rules is −0.017 R, 189 out of 480 are positive and the error on its mean is ±0.43. · 8 min
  3. 3How many tests you ranThe more rules you try, the better the best one looks, even if the data are noise. On 300 random series (the same candles shuffled in day blocks, with no trend) the best of a single rule is worth −0.03 R, the best of 20 is worth +0.16, the best of 480 is worth on average +0.28 R (95th percentile +0.43). On the real data the best is +0.74: above all 300 random series, but the honest comparison starts from +0.28 and not from zero. · 8 min
  4. 4Parameters and fittingEvery parameter left free is one more way to fit the rule to noise. On random series, choosing the best with 1 free parameter the mean R is +0.05 in the first part and +0.01 in the second; with 5 parameters it is +0.35 and +0.02. On the real data the rule chosen with 3-5 parameters goes from +0.75-0.82 R in the first part to +0.43-0.46 in the second: more than half of the gain, but with 25-27 trades the error is ±0.7. · 8 min

Part 2: In-sample and out-of-sample

In-sample and out-of-sample, walk-forward, looking into the future and costs.

  1. 5In-sample and out-of-sampleWe split the data in time: the first part (65%) to choose, the second (35%) to test. The rule chosen in the first part makes +0.82 ± 0.52 R (55 trades) and in the second +0.43 ± 0.73 (25): it loses half, and stays positive. The first decile of rules goes from +0.38 to +0.10 R, the last from −0.26 to −0.12: the advantage of the first part is kept only in part. On random series the best of the first part (+0.35) is worth +0.02 in the second. · 9 min
  2. 6Walk-forwardWalk-forward re-optimizes every month on the previous three months and trades the next month: every trade is out of sample by construction. On the 480 rules it gives 96 trades at +0.16 ± 0.33 R (+15.6 R) from March to September 2026; on random series the mean is +0.01 (95th percentile +0.27). With windows of 1, 2, 3 and 6 months the result goes from −0.13 to +0.45 R: the window length is one more parameter. · 9 min
  3. 7Looking into the futureA timing error turns a rule from −0.02 R into one of +0.83 R: the same 20-candle rule, entering at the open of the candle whose signal uses the close, wins 92% of the time and makes +342 R. A daily rule that uses the day's close to buy at the open gives +1.61 R per trade and 98% wins, against −0.15 R if you enter the next day. Numbers this good are the alarm signal. · 9 min
  4. 8Costs and execution in the backtestA backtest must decide what happens when a candle touches both stop and target, at what price you enter and how much it costs. The best rule of the grid does not change (+0.74 R in all cases), but a narrow-stop rule (10-candle breakout, 0.5 ATR stop, 1 R target) goes from −0.17 R with the prudent rule to +0.12 with the optimistic one: the sign depends on the assumption. With costs from 0 to 2 $ the grid mean goes from −0.001 to −0.110 R and the ranking of the rules stays almost the same (correlation 0.98). · 9 min

Part 3: Chance and robustness

The best of many attempts, the permutation test, parameter robustness and periods.

  1. 9The best of many attemptsThe p-value of a rule chosen beforehand is read against chance; that of the best of 480 must be read against the best that chance produces over 480. On 300 random series the best is worth on average +0.28 R (maximum +0.53); ours, +0.74, exceeds them all. But with random series that keep 10-day blocks the best is worth on average +0.53 (maximum +0.80) and ours exceeds 149 out of 150: the answer depends on what we call "chance". · 9 min
  2. 10The permutation testA permutation test shuffles part of the backtest and asks how often the shuffling does equally well or better than the real data. On the best rule (+0.74 R) random direction gives on average +0.23 (p-value 0.003), random timing +0.04 (p-value 0) and the shuffled series +0.05 (p-value 0). The median rule (−0.05) and the example rule (−0.02) stay in the middle of chance (p-values from 0.36 to 0.89). But the p-value of a single rule does not account for the 480. · 8 min
  3. 11Parameter robustnessA robust rule is worth almost the same if you change the parameters a little; a fragile rule is a spike. For the best (+0.74 R) the five neighbours (one parameter changed by one step) are worth from +0.28 to +0.66, on average +0.43: all positive, but at little more than half. The neighbours/best ratio is 0.58 against 0.52 for the best of the random series: it is not more robust than noise. Choosing by the mean of the neighbours did not help: +0.12 in the second part against +0.43. · 10 min
  4. 12Periods and regimesLet us split the ten months into five periods of about two months. The 240 breakout rules have a positive mean R in four periods out of five (from +0.02 to +0.22), the 240 fade rules are negative in all (from −0.19 to −0.02). The ranking of the rules is kept from one period to the next only at the start (rank correlation +0.72, +0.29, then −0.01 and +0.03). The best rule is positive in every period, but 48% of its gain comes from the first, when gold rises 18%. · 9 min

Part 4: Testing honestly

An honest testing process, mistakes, glossary and final test.

  1. 13An honest testing processWe put the tools together in a process written beforehand: family of rules fixed, data split, prudent execution assumptions, a selection criterion with p-value and neighbours, a single out-of-sample test. On our 480 rules the process chooses a rule (80-candle breakout, 2 ATR stop, 3 R target, duration 48: +0.53 ± 0.50 R, p-value 0.003) and in the single test it makes +0.12 ± 0.73 R over 23 trades: inconclusive. On random series the same procedure gives a positive test in 53% of cases. · 10 min
  2. 14A backtest told twice, mistakes, reference, glossary and final testThe same rule told in two ways: "+58 R, drawdown 6 R, profit factor 2.4" and "the best of 480, +0.74 ± 0.43 R, chance +0.28, out of sample +0.43 ± 0.73 over 25 trades, walk-forward +0.16 ± 0.33, protocol test +0.12 ± 0.73: interesting and unproven". Then the most frequent mistakes in chapter order, the quick reference with rules and measurements, the glossary and a final test of 15 questions. · 17 min

Another course

Forex course for beginners: from zero to your first demo tradeRisk and psychology in trading: the course that decides whether you lastMarket structure on gold: the book of swings, BOS and zones, with real chartsOrder blocks from A to Z on gold: the book of the zone, the displacement and the trade, with real chartsFVG and liquidity from A to Z on gold: the book of the price gap and the stop hunt, with real chartsSMC: the whole system on gold, from bias to order, tested step by step on real candlesIndicators from A to Z on gold: averages, RSI, MACD, Bollinger and the rest, measured on real candlesElliott and Fibonacci on gold: retracements, extensions and waves, measured on real candlesChart patterns on gold from A to Z: head and shoulders, double tops, triangles, wedges and flags, measured on real candlesCandlesticks on gold from A to Z: doji, hammer, engulfing, stars and soldiers, measured on real candlesHours, sessions and news on gold: when price moves, measured on real candlesSupport, resistance and levels from A to Z: how well they really hold, measured on real candlesRisk and position sizing from A to Z: simulations on real resultsPsychology and the trading journal from A to Z: measuring behavioural mistakesVolatility and ATR from A to Z: measuring how much gold movesComplete trading course: market structure, ICT, SMC and algorithmic tradingHow to follow the room signals: from message to trade, step by stepGold in depth: how XAUUSD moves and how to trade it with a methodPremium zone: scenario playbooks, chart exercises and trade reviews