Backtesting without illusions: costs, statistics, walk-forward and Monte Carlo

You have the signals: now you have to measure them. A well-made backtest is stricter than the average trader: it adds costs, assumes the least favourable case when it doesn't know, and returns numbers with margins of error. In this lesson you'll find the complete code (lib_backtest.py) and the control methods that separate an honest test from a pretty illusion.

The backtest engine (lib_backtest.py)

import numpy as np
import pandas as pd


def backtest(df, signals, stops, rr=2.0, pip=0.0001, spread_pips=1.0, buffer_pips=2.0, max_bars=48):
    """Long-only. Enters at the open of the candle after the signal.
    If both stop and target are touched in the same candle, the stop wins (prudent assumption).
    Returns a DataFrame with one trade per row and the outcome in R, costs included."""
    rows = []
    idx = df.index
    i = 0
    n = len(df)
    while i < n - 1:
        if signals.iloc[i] != 1:
            i += 1
            continue
        entry = df["open"].iloc[i + 1] + spread_pips * pip      # you buy at the ask
        stop = stops.iloc[i] - buffer_pips * pip
        risk = entry - stop
        if risk <= 0:
            i += 1
            continue
        target = entry + rr * risk
        outcome, exit_bar = None, None
        for j in range(i + 1, min(i + 1 + max_bars, n)):
            hit_stop = df["low"].iloc[j] <= stop
            hit_tp = df["high"].iloc[j] >= target
            if hit_stop:                     # caution: the stop wins when in doubt
                outcome, exit_bar = -1.0, j
                break
            if hit_tp:
                outcome, exit_bar = rr, j
                break
        if outcome is None:                  # time exit at the close of the last candle
            exit_bar = min(i + max_bars, n - 1)
            outcome = (df["close"].iloc[exit_bar] - entry) / risk
        rows.append({"time": idx[i + 1], "r": outcome, "risk_pips": risk / pip})
        i = exit_bar + 1                     # one position at a time
    return pd.DataFrame(rows, columns=["time", "r", "risk_pips"])


def stats(r):
    r = np.asarray(r, dtype=float)
    if len(r) == 0:
        return {}
    wins, losses = r[r > 0], r[r <= 0]
    equity = np.cumsum(r)
    peak = np.maximum.accumulate(equity)
    max_dd = (peak - equity).max()
    sd = r.std(ddof=1) if len(r) > 1 else float("nan")
    se = sd / np.sqrt(len(r)) if len(r) > 1 else float("nan")
    streak = cur = 0
    for x in r:
        cur = cur + 1 if x <= 0 else 0
        streak = max(streak, cur)
    return {
        "trades": len(r),
        "win_rate": len(wins) / len(r),
        "avg_win_R": wins.mean() if len(wins) else 0.0,
        "avg_loss_R": losses.mean() if len(losses) else 0.0,
        "expectancy_R": r.mean(),
        "ci95_R": (r.mean() - 1.96 * se, r.mean() + 1.96 * se),
        "profit_factor": wins.sum() / abs(losses.sum()) if losses.sum() != 0 else float("inf"),
        "max_drawdown_R": max_dd,
        "max_losing_streak": streak,
    }


def monte_carlo(r, n_sims=5000, seed=42):
    """Shuffles the order of the results in R and measures the maximum drawdown of each sequence."""
    rng = np.random.default_rng(seed)
    r = np.asarray(r, dtype=float)
    dds = np.empty(n_sims)
    finals = np.empty(n_sims)
    for k in range(n_sims):
        seq = rng.permutation(r)
        eq = np.cumsum(seq)
        dds[k] = (np.maximum.accumulate(eq) - eq).max()
        finals[k] = eq[-1]
    return {
        "dd_median": np.median(dds),
        "dd_p90": np.percentile(dds, 90),
        "dd_p95": np.percentile(dds, 95),
        "final_p5": np.percentile(finals, 5),
        "final_median": np.median(finals),
    }


def walk_forward(df, run, train_bars, test_bars):
    """Slides consecutive windows: 'run(df_window)' must return the window's results in R."""
    out = []
    start = 0
    while start + train_bars + test_bars <= len(df):
        test = df.iloc[start + train_bars : start + train_bars + test_bars]
        out.append(np.asarray(run(test), dtype=float))
        start += test_bars
    return out

Design choices (and why)

• Entry at the open of the candle after the signal, paying the spread (you buy at the ask).
• Stop = the sweep's low minus a margin, the risk is the entry-stop distance; the target is rr times the risk.
• Stop and target in the same candle → the stop wins: prudent assumption.
• Time exit (max_bars): if after a maximum number of candles nothing has happened, it closes at the close.
• One position at a time: avoids overlaps and unwanted combined risks.
• Results in R, costs included: the spread cost is already inside the entry.

A typical run

With a df of M5 candles and the signals from the previous lesson:

from lib_signals import bullish_signals
from lib_backtest import backtest, stats, monte_carlo

sig, stops = bullish_signals(df)
trades = backtest(df, sig, stops, rr=2.0, spread_pips=1.0, buffer_pips=2.0)
s = stats(trades["r"])
print(s["trades"], s["win_rate"], s["expectancy_R"], s["ci95_R"], s["max_drawdown_R"])
print(monte_carlo(trades["r"], n_sims=5000))

The result is a dictionary with number of trades, win rate, expectancy, 95% confidence interval, maximum drawdown and maximum losing streak. If the interval includes zero, you have no evidence of an edge.

Reading the statistics

The stats function computes what you saw in module 4: win rate, average R of wins and losses, expectancy, confidence interval (E ± 1.96 × SE), profit factor, maximum drawdown and consecutive losing streak. What to look at first:

1. Number of trades: if fewer than 100, everything else is unreliable.
2. Expectancy and interval: does the interval exclude zero?
3. Maximum drawdown and Monte Carlo (p95): is the real risk bearable?
4. Sensitivity to costs: recalculate with double spread: does expectancy stay positive?

The enemies of the backtest

Out-of-sample and walk-forward

Simple out-of-sample: you develop the rules on the first 60% of the data and verify them, without changes, on the remaining 40%.

Walk-forward: you slide consecutive windows through time: you use one window to "train" (define parameters) and the next to test, then move everything forward. It simulates how you'd really use the system: always with data it has never seen.

The walk_forward function returns the R results of each test window. Then you concatenate them and compute overall statistics:

from lib_backtest import walk_forward
import numpy as np

def run(df_window):
    sig, stops = bullish_signals(df_window)
    return backtest(df_window, sig, stops)["r"]

# 3 months of M5 = about 25,000 candles; test on 1 month at a time
results = walk_forward(df, run, train_bars=3 * 22 * 288, test_bars=1 * 22 * 288)
all_r = np.concatenate(results)
print(stats(all_r))

If the rules are fixed (no optimisation), walk-forward simply shows stability over time: does the system earn similarly in every window or depend on a single lucky period?

Parameter sensitivity

Change one parameter at a time within a small range (for example displacement 1.2 / 1.5 / 1.8; RR 1.5 / 2.0 / 3.0) and compare expectancy:

import itertools
for disp, rr in itertools.product([1.2, 1.5, 1.8], [1.5, 2.0, 3.0]):
    sig, stops = bullish_signals(df, disp=disp)
    tr = backtest(df, sig, stops, rr=rr)
    if len(tr) > 0:
        print(disp, rr, len(tr), round(stats(tr["r"])["expectancy_R"], 3))

What to look for: a plateau, that is nearby values with similar results. If only one combination "shines" and neighbours collapse, it's a fit to noise (an "isolated peak").

The control case (in code)

Compare your model with a simple control, same management: for example random signals in the killzone with the same frequency:

import numpy as np
import pandas as pd
from lib_structure import atr

rng = np.random.default_rng(0)

def random_signals(df, n_signals, atr_mult=1.5):
    a = atr(df, 14)
    idx = rng.choice(len(df) - 50, size=n_signals, replace=False)
    sig = pd.Series(0, index=df.index)
    stops = pd.Series(np.nan, index=df.index)
    for i in idx:
        sig.iloc[i] = 1
        stops.iloc[i] = df["close"].iloc[i] - atr_mult * a.iloc[i]
    return sig, stops

Repeat the random draw many times (for example 200) and you get the distribution of expectancy "by chance". Your model has an edge if its expectancy sits clearly beyond that distribution (for example above the 95th percentile).

Monte Carlo: what it tells you

The monte_carlo function shuffles the order of results and returns: median drawdown, drawdown at the 90th and 95th percentile and the final capital at the 5th percentile and median. Interpretation:

• dd_p95 is the drawdown you'll exceed only in 5% of cases: use it to choose risk per trade.
• final_p5 is the "unlucky but plausible" result at the end of the sample.

What this code does NOT do

• It doesn't handle swap, per-lot commissions and variable slippage.
• It doesn't optimise size (it's in R).
• It doesn't treat news and gaps specially.
• It's long-only and one position at a time.

These are deliberate limits to keep the code readable. Before trusting a result, extend the engine (for example with an hourly variable spread) and repeat the tests.

In short

• An honest backtest includes costs, prudent assumptions (stop before target) and entry at the next candle.
• Statistics must be read with the confidence interval: if it includes zero, there's no evidence.
• Validate with out-of-sample, walk-forward, parameter sensitivity (look for a plateau) and a random control.
• Monte Carlo gives you the realistic drawdown to use for choosing risk.
• Remember the model's limits and extend it before trusting it.

Practical exercise

1. Run the backtest on your dataset and compute statistics with the confidence interval.
2. Recalculate with double spread and with entry delayed by one candle: what changes?
3. Run the sensitivity grid and the random control (200 draws). Does your model exceed the 95th percentile of chance?