What overfitting actually is

Run a competition. Five hundred people each flip a coin ten times, and you keep the one who got the most heads. The winner will probably have flipped eight, nine, maybe ten heads out of ten.

Now look at that person’s record on its own, with the other 499 out of view. Nine heads from ten flips. If you did not know about the competition you would think they were extraordinary at flipping heads. You might pay them to flip for you. And when they flip for you they will get about five, because there was never anything to be good at.

The winner’s record was real. Nobody cheated. The record was still worthless as a prediction, because it was produced by selection rather than by skill, and selection does not repeat.

This is overfitting. In trading it usually happens like this: you have a strategy idea with a few settings. A moving average length. A stop distance. A time filter. You run the optimiser, it tries several hundred or several thousand combinations, and you keep the one with the best equity curve. That combination is your coin-flip winner.

Why it is so hard to see

The trouble is that the winning result looks exactly like a real edge. It has to. You selected it for looking good. Every property you would use to judge it, the smooth curve, the high win rate, the low drawdown, is the property you sorted on.

The information that would tell you whether it is real is not in the winning result at all. It is in the other 499. Specifically it is in how many you tried and how much better the winner was than the rest. If you tested four combinations and one was clearly best, that means something. If you tested four thousand and the winner beat the runner-up by a hair, it probably means you ran four thousand coin-flip competitions.

It is not only the optimiser

Optimiser runs are the obvious case because the software counts them for you. The subtler version is the searching you do yourself and do not count.

You test an idea and it does not work, so you add a filter. Still not quite, so you change the exit. Better, but there is a bad patch in the middle of 2022, so you add a volatility condition that skips it. Each of those decisions was a choice among alternatives, made by looking at the results. By the time you have a version you like, you have run a search. It just happened over three weeks in your head instead of in an afternoon on a server, so it does not feel like one and there is no counter in the corner of the screen.

This is the most common way people arrive at a strategy that cannot possibly work, while being completely honest at every individual step.

What honest correction looks like

There is published statistics for this. The general shape is that you take the performance you observed, ask how good the best of that many random attempts would have looked anyway, and keep only the part that exceeds it. The version this site runs is written up in the probability of backtest overfitting, if you want the algorithm rather than the idea.

The uncomfortable arithmetic is that the bar rises with the number of attempts. Search harder and you need a better result to clear it, because searching harder is precisely what makes an impressive result cheap to obtain. A strategy selected from four thousand combinations on a few hundred trades often does not survive this at all, and that verdict is usually correct.

Two things follow, and they are the practical content of this lesson. Count your variations and write the number down as you go, because reconstructing it later is guesswork and the guess is always low. And keep the results of the runs you discarded, because the spread of results across everything you tried is the evidence that tells you whether the winner stood out or merely came first. Both habits in practice, on a real edge, are written up in eleven pre-registered improvements of which ten made things worse.

The next lesson is about the thing that kills more candidate strategies than overfitting does, and it needs no statistics at all: what costs do to an edge.