Why calibrate at all
Every test OverfitCheck runs is implemented from a peer-reviewed paper and checked against the numerical examples in that paper (see Research & papers). That proves the math is right. It does not prove the product judges well. For that you need trade logs where the correct answer is known independently of the engine. Synthetic data cannot give you one, because whoever generates the data also decides what the answer should be.
Our own multi-year systematic trading research program had exactly that on hand. It produced a family of real strategy variants on a Nasdaq-100 CFD at a major CFD broker. Years of testing, forensics, and live trading settled which ones were real, which were marginal, and which deserved to die. Those verdicts were fixed long before this engine existed. So the setup was clean. Give the engine nothing but the trade logs and see whether it lands on the same verdicts.
Protocol
Six MetaTrader 5 Strategy Tester reports went through the identical code path a paying user hits. M1 backtests spanning 2012–2025, 164–1,111 trades each. No special-casing, no tuning, no access to the ground truth.
MT5 HTML report → Deals-table extraction. On all six reports the summed per-trade P&L reconciled with the report's own balance change to the cent, and the universal CSV parser fuzzy-matched all 9 columns correctly (see MT5 report format).
All 8 statistical tests with the exact upload-pipeline defaults. Honest single-configuration trial count (n_trials=1), futures/CFD cost model, and full bootstrap confidence intervals at 500 resamples per test, 200 for Monte Carlo.
The standard score penalty table, the same one documented on The Tests, with no adjustments. Every audit finished in under 30 seconds, 12.6s to 23.2s including all confidence intervals.
Two requirements were written down before the run. The losing killed strategy must grade at or near F. The live incumbent must outscore every killed variant. Failing either meant fixing the engine before launch, not softening the writeup.
Results
| Strategy (anonymized) | Ground truth | Trades | PF | Score | Penalties |
|---|---|---|---|---|---|
| Live incumbent | Validated edge; traded live | 867 | 1.30 | 60 / B | PBO −20 · Regime −20 |
| Clock-corrected variant | The incumbent rebuilt on a corrected server clock; real | 898 | 1.28 | 60 / B | PBO −20 · Regime −20 |
| Timeframe-degraded variant | The incumbent on a coarser timeframe; real but weaker | 1,111 | 1.18 | 60 / B | PBO −20 · Regime −20 |
| Retired variant | Real edge that decayed out-of-sample; retired | 581 | 1.24 | 45 / C | PBO −20 · WF −15 · Regime −20 |
| Killed fade | Marginal; killed by the research program | 284 | 1.11 | 0 / F | PBO −20 · WF −35 · MC −25 · Regime −20 · DSR −20 |
| Killed strategy | Losing; killed by the research program | 164 | 0.62 | 0 / F | PBO −20 · WF −35 · MC −25 · Regime −20 · DSR −20 |
Both pre-registered requirements passed. The losing strategy scored 0/F. The incumbent scored 60/B and beat both killed variants. What happens between those two extremes is the part worth reading.
A profit factor of 1.11 looks tradeable in a spreadsheet. The engine disagreed, on the same grounds the research program had: no statistically significant edge (Monte Carlo p = 0.27), an out-of-sample Sharpe collapse to −0.8, and a 37% transaction-cost haircut. Marginal edges die on costs. The engine saw it from the trade log alone.
Real edge, real decay: 45/C, sitting between the incumbent (B) and the killed pair (F). That is precisely its ground-truth position. Its full audit is public as the second card on the demo page.
It penalized the incumbent's regime concentration (−20). That is not a false positive. The research program had independently documented heavy year-and-season profit concentration in these strategies. The engine found a real property of a real live edge, unprompted.
The rank-order check
Grades are coarse, so the sharper test is ordering. Sort the six strategies by the engine's continuous outputs, either the Monte Carlo p-value or the Probabilistic Sharpe Ratio, and the ground-truth quality order comes back exactly:
| Ground-truth rank | Strategy | MC p-value | PSR |
|---|---|---|---|
| 1 | Live incumbent | 0.0004 | 0.9996 |
| 2 | Clock-corrected variant | 0.0014 | 0.9993 |
| 3 | Timeframe-degraded variant | 0.0139 | 0.990 |
| 4 | Retired variant | 0.0148 | 0.988 |
| 5 | Killed fade | 0.270 | 0.742 |
| 6 | Killed strategy | 0.950 | 0.090 |
Worth noting what this ordering is not. It is not profit-factor order. In ground truth the timeframe-degraded variant (PF 1.18) sat at least level with the retired variant (PF 1.24), because the retired variant's edge had decayed harder despite the better profit factor. The engine's continuous outputs placed the pair the same way, separated only marginally (p 0.0139 vs 0.0148, PSR 0.990 vs 0.988). An engine that just dressed profit factor up in statistics would have put the retired variant clearly ahead. Ranks 3 and 4 are the near-tie in this table. Every other adjacent pair is separated decisively.
Reproducibility note. The engine uses unseeded random number generators, so Monte Carlo, PBO-proxy, and confidence-interval values wobble in the third decimal from run to run. Grades and penalties are stable.
What the calibration exposed
First the scope. Six fixtures is a small set, and all six come from one research program on one instrument family. The run shows the engine ranked these six correctly. It does not establish how the engine behaves on strategy types this set does not contain. Read it as one hard test passed, not as coverage.
Then the findings. A validation you can trust is one that reports what it found wrong. Five follow, and all five are disclosed in the product. Where a fix would silently shift every historical score, we left the behavior visible instead of quietly patching it.
When you upload one configuration, full CSCV cannot run and a bootstrap proxy substitutes. Calibration showed the proxy sits at ≈0.5 for any strategy by construction. All six fixtures, live edge and killed alike, scored between 0.467 and 0.501. Zero discriminative power. Under the current penalty table this applies a uniform −20 to every single-configuration upload, capping such scores at 80. It did not distort the calibration ranking, because it hit all six equally, and multi-configuration CSCV mode is unaffected. Treat the single-strategy PBO figure as an advisory, not as evidence. The results page says so where it applies. See PBO & CSCV.
The Minimum Backtest Length formula scales with the number of tested configurations. At an honestly-reported single trial it requires zero trades, so even the losing strategy “passed” sample size. That is the paper's formula, not a bug. It does mean the test only protects you if you report your true trial count. Understate it and two tests, Sample Size and Deflated Sharpe, both go too easy on you.
The timeframe-degraded variant tied the incumbent at 60/B even though the continuous test outputs separate them clearly. Both landed in the same penalty bands. Finer bands would separate them, at the cost of the simple, fully documented penalty table. We kept the documented table and disclose the tie.
The top-quartile concentration penalty triggers above 70%, so the incumbent's 71% was penalized identically to the killed strategy's 99%. The flag itself was honest in every case. Past the threshold, though, the band cannot tell moderate concentration from extreme concentration.
When a strategy loses money in-sample, the walk-forward degradation ratio is forced to zero, and the penalty text wrongly blamed “severe OOS degradation”. That was misleading when out-of-sample was the good period. The penalty was correct. The explanation was not. Fixed during this step. No scores changed.
What we do not disclose
The incumbent is a live strategy. So we publish everything about the engine's behavior: scores, penalties, test values, failure modes. We publish nothing that identifies the strategies. No entry logic, no parameters, no session windows, no exact instrument symbol, no broker name. “A Nasdaq-100 CFD at a major CFD broker” is as specific as this page will ever get. The same anonymization rules govern the blog's research posts, which tell the stories behind several of these fixtures.