Backtesting without illusions: costs, overfitting and walk-forward

A minimal backtest

The code below enters at the open of the candle after the signal, uses fixed stops and targets in pips and, if both are touched in the same candle, cautiously assumes the stop was hit first. The spread cost is subtracted in R.

import numpy as np

def backtest(df, signals, stop_pips, rr=2.0, pip=0.0001, spread_pips=0.8):
    """signals: Series with +1 (long), -1 (short), 0 (no signal)."""
    results = []
    i, n = 0, len(df)
    while i < n - 1:
        s = signals.iloc[i]
        if s == 0:
            i += 1
            continue
        entry = df["open"].iloc[i + 1]
        risk = stop_pips * pip
        stop = entry - s * risk
        target = entry + s * risk * rr
        outcome, j = None, i + 1
        while j < n:
            hi, lo = df["high"].iloc[j], df["low"].iloc[j]
            hit_stop = lo <= stop if s == 1 else hi >= stop
            hit_tp = hi >= target if s == 1 else lo <= target
            if hit_stop:              # caution: the stop wins when in doubt
                outcome = -1.0
                break
            if hit_tp:
                outcome = rr
                break
            j += 1
        if outcome is None:
            break
        results.append(outcome - spread_pips / stop_pips)   # cost in R
        i = j + 1
    return np.array(results)

Statistics

def stats(r):
    wins, losses = r[r > 0], r[r <= 0]
    equity = np.cumsum(r)
    drawdown = (np.maximum.accumulate(equity) - equity).max()
    return {
        "trades": len(r),
        "win_rate": len(wins) / len(r),
        "expectancy_R": r.mean(),
        "profit_factor": wins.sum() / abs(losses.sum()) if len(losses) else float("inf"),
        "max_drawdown_R": drawdown,
    }

The enemies of the backtest

• Look-ahead: using in a candle information you only know afterwards (like unconfirmed swings)
• Overfitting: trying 200 parameter combinations and keeping the best. The best is almost always luck
• Costs: spread, commissions, slippage and swap. Test with costs higher than expected
• Data quality: gaps, shifted hours, differences between brokers
• Sample: few trades prove nothing

Control techniques

• In-sample and out-of-sample: develop on one part of the data, verify on the other, never looked at
• Walk-forward: optimise on one window, test on the next, and slide the windows through time
• Parameter sensitivity: if changing n from 3 to 4 makes the result collapse, the model is fragile
• Testing on different instruments and periods, without changing the rules
• Monte Carlo on trade order to estimate possible drawdowns

Golden rule

If the result looks too good to be true, there is probably an error in the test. Look for the bug before believing the result.