Multiple testing and selection

Updated 2026-09-20

Definition

The fact that, by trying many rules on the same data and keeping the best, the result of the best overstates what awaits the rule on new data.

Why it matters

It grows with the number of trials: on random series the best of one rule is worth −0.03 R, of 20 is worth +0.16, of 480 is worth +0.28 on average. All trials must be counted, hidden ones included (timeframes, filters, versions).

Practical example

On 300 random series the best of 480 rules has a mean R of +0.28 (maximum +0.53); in one series out of three it exceeds +0.3.

Learn more

Course lesson: How many tests you ran

Related entries

Want to compare notes with other traders?

Join the community to share ideas and questions, or the free signal room on Telegram.

← Full glossary

Educational content, not financial advice. Trading involves risk.