Part 3: Chance and robustness · Chapter 12 of 14 · 9 min
Periods and regimes
Let us split the ten months into five periods of about two months. The 240 breakout rules have a positive mean R in four periods out of five (from +0.02 to +0.22), the 240 fade rules are negative in all (from −0.19 to −0.02). The ranking of the rules is kept from one period to the next only at the start (rank correlation +0.72, +0.29, then −0.01 and +0.03). The best rule is positive in every period, but 48% of its gain comes from the first, when gold rises 18%.
The market changes
A ten-month backtest is a photograph of a stretch of market, not of "the market". In ten months gold had a strong rise (at the start), stretches of long decline and stretches almost flat. These stretches are called regimes: periods in which the market behaves in a certain way (trend, range, high or low volatility). A rule that follows the movements earns in trending regimes and loses in directionless ones, and the opposite holds for one that goes against them. A backtest that crosses only one type of regime measures the rule and the regime together, and they cannot be separated.
The simplest check: look at the result period by period.
Five periods
We split the 4,865 candles into five periods of equal length (about 973 candles, two months). For each: the movement of gold and the mean R of the 240 breakout rules and the 240 fade rules (mean over the rules), the number of positive rules and the best rule of chapter 2.
| Period | Gold in the period | Breakout: mean R | Fade: mean R | Positive rules | The best of chapter 2: trades, mean R, sum |
|---|---|---|---|---|---|
| 1 (15 Dec - 11 Feb) | +17.8% | +0.22 | −0.19 | 194 out of 480 | 16, +1.66, +26.5 R |
| 2 (11 Feb - 10 Apr) | −5.6% | +0.06 | −0.18 | 179 | 17, +0.54, +9.2 R |
| 3 (10 Apr - 9 Jun) | −10.7% | +0.02 | −0.06 | 226 | 17, +0.21, +3.6 R |
| 4 (9 Jun - 7 Aug) | +0.1% | +0.17 | −0.19 | 238 | 14, +0.72, +10.1 R |
| 5 (7 Aug - 6 Oct) | −2.5% | −0.02 | −0.02 | 215 | 18, +0.32, +5.8 R |
What it reads.
- Fade is negative in all periods (from −0.19 to −0.02). In the ten months going against the movements always cost.
- Breakout is positive in four periods out of five, but the size changes a lot: +0.22 in the first (gold rises 18%) and +0.17 in the fourth, close to zero in the second and third, −0.02 in the fifth. The edge of breakouts does not coincide with the direction of gold: in the fourth period the price ends where it started and breakouts earn all the same. What explains it, with these data we cannot say.
- The best rule is positive in every period, but the sum is very different: +26.5 R in the first, +9.2, +3.6, +10.1 and +5.8 in the others. The first period makes 48% of the total; without it the rule is worth about +28.7 R over four periods.
Read the chart in words
Daily gold chart with, below, two curves of mean cumulative R per rule: that of breakouts rising, that of fades falling steadily.
The ranking is not kept
A rule is worth something if, good in one period, it is also good in the next. Let us compute the rank correlation between the results of the 480 rules in consecutive periods:
| Periods compared | Rank correlation |
|---|---|
| 1 and 2 | +0.72 |
| 2 and 3 | +0.29 |
| 3 and 4 | −0.01 |
| 4 and 5 | +0.03 |
At the start the ranking is kept (+0.72), then it is lost: from the third period on, which rules do well in one period says almost nothing about which do well in the next. Part of this is real structure that changes (the regime), and part is noise (with 14-18 trades per period, the mean R of a rule has an error of ±0.8 or more). In practice: a two-month result says little about the following two months.
Where the gain comes from, month by month
The best rule, month by month (R of the trades closed in that month):
Read the chart in words
Daily gold chart with, below, ten bars of the monthly R of the best rule, mostly positive.
Seven months out of ten are positive, three near zero or negative (April, July, September). Two months, January (+15.6) and August (+14.1), make +29.7 R by themselves, more than half the total. This is typical of rules that aim at a large target (3 R): they lose more than half the time (they win 46%, with an average loss of 0.99 R) and now and then earn a lot (average win 2.79 R). The gain depends on a few big moves, and if none occurs, the rule can spend months near zero.
What it means for the future
What can we say, with these numbers, and what not?
- We can say that in the ten months following the movements paid and going against them did not, with some variation, and that the edge of breakouts was positive in most periods.
- We cannot say that it will continue. We have seen a single stretch of market (a strong rise, a long decline, stretches of wide swings), and we do not know what the same rules would do in a year of narrow range and low volatility, or in a phase with a long opposite trend.
- We cannot separate the rule from the regime. To do so you would need more years, more instruments, or different regimes.
This is the largest limit of the book, and it holds for every backtest: the sample is a stretch of history. The shorter the sample, the more it resembles a single regime.
What to do with it
- Always look at the result by period (two, three months) and by month: where does the gain come from?
- Remove the best period and see what remains.
- Ask in which regime you have been: trend, range, high or low volatility, and how many different regimes there are in the sample.
- Do not count nearby periods as independent trials: the correlation between periods says how much new information there is.
- Try the rule on other periods and other instruments, when you can, before believing it.
Procedure in four steps
- Split the sample into 4-6 equal periods and compute the result of each.
- Remove the best period and recompute.
- Look at the result month by month and how many months make half the total.
- Declare which regimes the sample contains and which it does not.
Exercises
Chapter exercises
1. The best rule has sums of +26.5, +9.2, +3.6, +10.1 and +5.8 R in the five periods (total +55.2 R over the periods). What share comes from the first period?
2. Fade is negative in all five periods and breakout is positive in four. How is it read?
3. The rank correlation of the ranking of rules between consecutive periods is +0.72, +0.29, −0.01, +0.03. What does it mean?
4. Why are five periods of two months a weak test of stability?
Have a question or want to share your exercise?
Post in the community, or join the free signals room on Telegram.
Educational content, not financial advice. Trading involves risk.