Why this matters
Every parameter you test counts as a trial, whether or not you thought of it that way. A moving average length. A stop-loss distance. An entry filter threshold. Test 50 combinations and keep the best performer, and you have almost certainly kept the one that fit the random noise in that one dataset, not the one with a repeatable edge.
Statisticians call this the multiple-testing problem. It is the main reason backtested strategies underperform live. Results that looked significant on one dataset fall apart as soon as the data changes. Standard backtesting software does not measure this. OverfitCheck does.
How CSCV works
The test cuts your history into chunks, then checks every possible way of splitting those chunks into a tuning half and a testing half. The formal name is Combinatorially Symmetric Cross-Validation, or CSCV, introduced by Bailey, Borwein, López de Prado & Zhu (2015). Five steps:
The trade log is divided into N = 16 equal time-ordered subsets. Using 16 subsets gives C(16, 8) = 12,870 combinations. That is enough for a stable distribution of outcomes while staying fast to compute.
All C(16, 8) = 12,870 ways to assign 8 subsets to in-sample (IS) and 8 to out-of-sample (OOS) are listed out. Checking every split is what makes the test hard to fool. No single arbitrary IS/OOS boundary decides the result.
For each of the 12,870 splits, the best-performing strategy configuration on the IS periods is selected. When only a single configuration is uploaded, a bootstrap proxy is used instead. See below.
The IS winner's rank among all configurations on the OOS periods is recorded, then turned into a single number, λ, with λ = log(r/S ÷ (1 − r/S)), where r is the OOS rank and S is the number of configurations. A negative λ means the IS winner ranked in the bottom half out of sample.
PBO = the fraction of all 12,870 splits where λ < 0. In plain words, the fraction of splits where the IS winner ranked in the bottom half of OOS performance. A PBO of 0.5 means the IS winner underperformed in exactly half of all splits. A PBO of 0.8 means it underperformed in 80% of them.
Reading your PBO score
The IS winner also came out ahead out of sample in most splits. The strategy holds up on data it was not tuned on.
In-sample and out-of-sample results agree about as often as a coin flip. That points to partial overfitting. Cut free parameters.
The IS winner fell behind in most OOS splits. What the backtest found is more likely tuned noise than a real edge.
PBO carries the heaviest penalty in the score system (−35 points) because overfitting is the biggest single threat to live performance. Read a high PBO carefully. It does not mean the strategy loses money. It means the backtest cannot be trusted as evidence either way.
Single-strategy mode
Full CSCV needs multiple strategy configurations to compare. When you upload a single backtest export, OverfitCheck uses a bootstrap proxy instead. It resamples your trade returns 10,000 times with replacement and computes the Sharpe ratio on each bootstrap sample.
The proxy score is the fraction of bootstrap samples with a Sharpe at least as high as your actual sequence. Now the honest problem with it. The bootstrap distribution is built by resampling your own trades, so it centers on your own observed Sharpe, and the proxy comes out near 0.5 for any strategy whatsoever. In our ground-truth calibration, all six real strategies scored between 0.467 and 0.501. A validated live edge and two killed strategies alike. In single-strategy mode this test tells you nothing about your strategy in particular.
Treat the single-strategy PBO figure as an advisory, not as evidence for or against overfitting. Under the current penalty table its ≈0.5 value puts nearly every single-configuration upload in the 0.4–0.6 band (−20 points), which caps single-upload scores at 80. The cap hits everyone equally, so comparing your own strategies against each other still works. Your results page says when the proxy was used. And the question the proxy was aiming at, whether your average trade is really above zero, is already answered properly by the Monte Carlo permutation test.
Upload exports from multiple parameter configurations in the same file. OverfitCheck then runs full CSCV across all configurations, replacing the bootstrap proxy with the complete 12,870-split algorithm described above. CSCV mode is unaffected by this limitation.
Improving a high PBO score
Every knob you can turn is another trial. A strategy with 2 parameters and 30 test runs is far less exposed to overfitting than one with 5 parameters and 500 test runs. Fewer knobs, less to overfit.
Pick your parameters on the in-sample data, write them down, then run them once on a period you held back. Never tune parameters on the same data you use to judge them.
Start from something you noticed in the market, say “momentum persists after earnings gaps”, instead of hunting for a pretty equity curve. A strategy that starts from an idea about how the market works is more likely to be real than one you found by searching.