EA backtest overfitting

EA Backtest Traps: Overfitting, Look-Ahead Bias, and How to Prove It

The six ways an EA backtest lies to you, the statistics that detect them, and a Deflated Sharpe Ratio calculator with numbers you can reproduce.

By Auriatech Research Desk · · Last updated 2026-11-08 · 3 min read
backtestingstatisticsEAmethodology

A backtest is a claim, not a result. It says: if this rule had been running under these conditions, this is what would have happened. The claim is only as strong as the conditions you actually reproduced — the data, the spread, the fill model, and the number of times you tried variations before you picked the one you are showing people.

This guide covers the six ways an Expert Advisor backtest lies, the statistics that detect them, and a Deflated Sharpe Ratio calculator you can run yourself.

Why “it looked good on the tester” is not evidence

The core problem is that financial strategies have a low signal-to-noise ratio. The signal is weak, so the noise is large relative to it, and a search over many configurations reliably finds a configuration that fits the noise.

Bailey, Borwein, López de Prado and Zhu state this precisely in The Probability of Backtest Overfitting: a 5% false-positive rate only holds if you apply a test exactly once. In practice you apply it on the same data repeatedly, “making the emergence of false positives almost certain.”

They note the scale of the problem for even a trivial strategy. For a moving average crossover with two sample lengths, an entry threshold, an exit threshold and a stop-loss, the number of parameter combinations “over thousands of securities is in the billions.” Each combination produces a Sharpe ratio. Each Sharpe ratio gets a significance test. Each test carries a 5% false-positive rate. That is not a subtle edge — it is a machine for producing convincing nonsense.

Two structural features of finance make it worse than the natural sciences:

  1. Profitable strategies are rare because there is intense competition for them, which drives the signal-to-noise ratio down.
  2. True findings are short-lived because financial systems adapt.
  3. Controlled experiments are essentially impossible, so a false claim usually cannot be debunked by replication — it just persists.

The practical consequence for an EA developer: the number of trials you ran is part of your result. A backtest that does not report N — the number of configurations tried before the winner was selected — cannot be assessed at all, because the hold-out method ignores N entirely.

Trap 1: Multiple testing / selection bias

This is the big one, and it has a formula attached.

The Deflated Sharpe Ratio (DSR) answers a precise question: given that I tested N configurations and picked the best one, what is the probability that the selected strategy’s Sharpe ratio is genuinely positive?

It is a Probabilistic Sharpe Ratio (PSR) in which the benchmark has been raised from zero to the expected maximum Sharpe ratio among N zero-skill trials.

The formulas

Let s be the per-period Sharpe ratio of the selected strategy, T the number of return observations, γ₃ the sample skewness, γ₄ the Pearson kurtosis (3 for a Gaussian), σ_trials the cross-sectional standard deviation of the per-period Sharpe estimates across the N trials, and γ ≈ 0.57721566 the Euler–Mascheroni constant.

maxZ    = (1 − γ)·Φ⁻¹(1 − 1/N) + γ·Φ⁻¹(1 − 1/(N·e))

SR₀     = σ_trials · maxZ

DSR     = Φ( ((s − SR₀)·√(T − 1)) / √(1 − γ₃·s + ((γ₄ − 1)/4)·s²) )

Where Φ is the standard normal CDF and Φ⁻¹ its inverse.

The intuition: SR₀ is the Sharpe ratio you would expect the best of N random guesses to have. If your selected strategy’s Sharpe is barely above SR₀, then something as simple as luck explains it, and the DSR says so.

Why SR₀ grows with N

maxZ(N) is the expected maximum of N standard normal draws. More trials means a higher bar:

N (trials) maxZ(N)
10 1.575
50 2.276
100 2.531
200 2.766
500 3.053
1,000 3.255

Note that maxZ grows only with the logarithm of N — so going from 100 trials to 10,000 trials raises the bar by about 0.7 standard deviations of a Sharpe estimate. The penalty is real but slow. What kills you is not the slow growth of the bar; it is that SR₀ scales with σ_trials, and in real backtests the trials are often far more correlated than the formula assumes.

Worked numbers (reproducible)

Assumptions: daily returns, 252 trading days/year, normal returns (skew 0, Pearson kurtosis 3), and σ_trials = 1/√(T − 1), which is the standard approximation for the dispersion of sample Sharpe ratios under a zero true Sharpe with iid trials.

Case 1 — How the number of trials changes the verdict.

Observed annualised Sharpe = 1.50, over 756 daily observations (3 years).

N (trials) SR₀ (annualised) z-score DSR
10 0.910 1.019 0.846
50 1.315 0.319 0.625
100 1.462 0.066 0.526
500 1.764 −0.455 0.325
1,000 1.881 −0.657 0.256

A Sharpe of 1.50 that looks convincing on a chart gives a DSR of 0.526 if you tried 100 configurations — meaning roughly a coin flip that the edge is real rather than the best of 100 lucky draws. At 1,000 trials the DSR falls to 0.256.

Case 2 — How the observed Sharpe changes the verdict (N = 100, T = 756):

Annualised Sharpe SR₀ (ann.) z DSR
0.50 1.462 −1.665 0.048
0.75 1.462 −1.232 0.109
1.00 1.462 −0.799 0.212
1.25 1.462 −0.366 0.357
1.50 1.462 0.066 0.526
2.00 1.462 0.928 0.823
2.50 1.462 1.786 0.963

Case 3 — How the track record length changes the verdict (N = 100, annualised Sharpe 1.50):

Length T (obs) SR₀ (ann.) z DSR
1 year 252 2.536 −1.031 0.151
2 years 504 1.791 −0.410 0.341
3 years 756 1.462 0.066 0.526
6 years 1,512 1.033 1.140 0.873
12 years 3,024 0.731 2.659 0.996

Case 4 — How non-normality changes the verdict (N = 100, annualised Sharpe 1.50):

Skew γ₃ Pearson kurtosis γ₄ z DSR
0 3 0.066 0.526
−1 6 0.063 0.525
−3 15 0.057 0.523
−1 6 (at ann. SR 2.50) 1.648 0.950
−3 15 (at ann. SR 2.50) 1.439 0.925

The skew and kurtosis terms are small at moderate Sharpe ratios and only bite hard at high ones. This matters for grid strategies specifically: a grid’s return distribution is famously negatively skewed with fat tails — many small wins punctuated by occasional large losses. At an annualised Sharpe of 2.50, moving from Gaussian assumptions to (skew −1, kurtosis 6) drops the DSR from 0.963 to 0.950 — i.e. straight through the significance threshold.

The punchline in one table

Annualised Sharpe Max trials that still passes DSR ≥ 0.95
1.00 ~2 (3 yrs) / ~50 (12 yrs)
1.50 ~5 (3 yrs) / ~50 (6 yrs) / ~10,000 (12 yrs)
2.00 ~50 (3 yrs) / ~5,000 (6 yrs)

An annualised Sharpe of 1.5 over three years survives about five configurations. If you ran a parameter sweep with 200 combinations and picked the best, a Sharpe of 1.5 is not evidence of an edge — it is close to the arithmetic expectation.

This is why you must report N. A Sharpe ratio without the trial count is not a result; it is a number with the interesting part removed.

Trap 2: Look-ahead bias

Look-ahead bias is any mechanism by which the backtest uses information that would not have been available at the moment the trade was supposedly taken.

On MT5 specifically, the common sources are:

  • Using the current bar’s close to make a decision that is filled at the current bar’s open. If your EA checks a crossover condition and opens on the same bar, it is assuming it knew the close before it printed.
  • Repainting indicators. Any indicator that recalculates its history (painting the previous bar, shifting values) makes the past look better than it was tradeable.
  • iBarShift-style off-by-one errors. Shifting by 0 instead of 1 (or using a future bar index) leaks.
  • Using revised economic data. CPI, NFP and employment figures are revised. A backtest that reads the final revised print on the release date is trading on information that did not exist yet.
  • Using the final, back-filled history available from the broker. The broker’s server history is not always the same as what a client saw at the time. Custom symbol imports make this worse, not better, if you import history that was assembled after the fact.

The test: for every input your EA reads, ask “at the moment it fired, was this value final?” If the answer is “it was final later”, you have look-ahead.

A concrete example of the cost. A rule that enters on the close of a bar whose high exceeded the previous bar’s high is a legitimate breakout rule if the entry happens on the next bar’s open. The same rule entered at the close of the signal bar is look-ahead — and it will look spectacular.

Trap 3: Understated costs

A backtest that assumes a flat spread is a backtest of a strategy that does not exist. On gold the spread is not a constant: it widens at session boundaries, around scheduled US releases, and during thin Asian hours.

Three costs that must be in the model:

  1. Spread, ideally variable, not flat.
  2. Commission, if your broker charges it (per lot, per side).
  3. Swap/financing, if you hold positions overnight — and on a grid, you hold them for a long time. Three days of negative swap on a multi-layer gold grid is a real, recurring drag that a “close at end of test” backtest often omits.

MT5’s strategy tester lets you set spread explicitly, and the documentation is explicit that a fixed spread should be set at or above your broker’s typical spread for the session you trade — the spread current at the moment you ran the test is not representative of anything.

The rule: if a strategy only works at a spread of $0.10 on gold, it does not work. State the spread assumption prominently, then state it again in the results table.

Trap 4: Tick modelling that flatters the strategy

MT5 offers several tick generation modes and they do not agree. From MetaQuotes’ own documentation, in a test of one simple strategy over the same history:

Parameter 1 minute OHLC Every tick Every tick based on real ticks
Ticks modelled 731,466 18,983,485 34,099,141
Testing time incl. tick generation 1.6 s 36.7 s 74 s

The MetaQuotes article notes the practical consequence directly: in “Every tick” mode you may receive a monotonically increasing or decreasing sequence of ticks that “virtually guarantees a market entry after the range is broken through.” That is a mechanical artefact, not market behaviour — and it systematically flatters breakout and stop-entry strategies.

Guidance that follows from the platform documentation:

  • Open prices only — fastest, least accurate. EA only acts on a new bar’s open. Good for a first sweep, never for a verdict.
  • 1 minute OHLC — good for rules that act on bar close.
  • Every tick — reasonable default for most systems.
  • Every tick based on real ticks — the accuracy benchmark. Use it for the final run and for anything with tight stops or intrabar entries.

And if you import your own history as a custom symbol, MetaQuotes documents that the price history is stored in a separate Custom directory and — importantly — that the tester downloads tick data from the broker in real-ticks mode, with the caveat that where real ticks are absent for some intervals the tester simulates the price like “Every tick” mode. Check the history quality figure in the test report. If it is materially below 100%, your run contains simulated segments and you should say so.

The specific danger for gold: a grid EA with stops set a few dollars away trades the intrabar path, and the intrabar path is exactly what tick modelling invents.

Trap 5: Holding out a “test set” you already contaminated

The hold-out method is the most widely used validation approach and Bailey et al. explain why it is inadequate:

  1. If the data is publicly available, the researcher probably used the hold-out as part of the in-sample set, at least unconsciously.
  2. Any seasoned researcher knows how the variables performed over the hold-out period, and may have used that knowledge in the design.
  3. For small samples it is badly underpowered. Weiss and Kulikowski argue hold-out should not be applied to an analysis with fewer than 1,000 observations; for a weekly strategy that means a backtest of under 20 years.
  4. It consumes data you need for design. Taking the OOS set from the end of the series loses the most recent, most representative observations.
  5. It ignores N. As long as you tried more than one configuration, overfitting is present — the hold-out method has no way to know how much.

What to use instead: walk-forward with a built-in forward period

MT5’s Strategy Tester has this natively. From the platform documentation, the Forward option divides the selected period into an optimisation period and a forward period (a half, a third, a quarter, or a custom start date). Optimisation runs on the first period. Then 10% of the best runs (full search) or 25% (genetic algorithm) are selected and re-tested on the forward period, with both results shown side by side on the Forward Results tab. The forward period is always the later part of the range.

The MetaQuotes guidance is that if the results on both periods coincide, “the more likely it is that the Expert Advisor will show good results in real trading.”

That is the correct read, with one caveat: a parameter set that ranks well in both periods is a candidate; one that ranks well only in sample is a fit. And you should still report the total trial count N, because the forward test does not eliminate selection bias — it only measures how much of it generalises in one particular split.

Trap 6: Survivorship and regime blindness

Two more that are easy to miss:

Survivorship bias in the instrument. If you only test XAUUSD because it is the instrument that worked, you have selected your data. A strategy that works on gold and fails on silver and EURUSD is not a “gold strategy” — it is a strategy that happened to fit one instrument’s history.

Regime blindness. Most backtests run over one contiguous period. If that period contains only one volatility regime, the result tells you nothing about the regime you will actually trade. On gold, the relevant regimes are roughly:

  • 2011–2015: a long post-crisis drift with sharp drawdowns
  • 2016–2019: low-volatility, range-bound, algorithmically traded
  • March 2020: a liquidity dislocation — gaps through stops, spreads at levels that did not exist for years
  • 2022: a sustained rate-driven trend, the cleanest trend gold has seen
  • 2024–2026: an unusually strong uptrend with large central-bank-driven pullbacks

If a backtest covers one of these and you draw a conclusion about “gold”, your conclusion is about that regime.

A validation checklist you can actually run

Before you publish or fund any EA result:

  1. State N — the total number of configurations tried, including the discarded ones.
  2. Compute the DSR using the formulas above. If DSR < 0.95, do not present the Sharpe ratio as evidence of an edge.
  3. Run a forward/walk-forward split and compare in-sample and forward rankings, not just the headline numbers.
  4. Re-run at a spread at or above your broker’s typical spread for the sessions you trade, with commission and swap included.
  5. Re-run with “Every tick based on real ticks” and report the history quality percentage.
  6. Test across at least two different volatility regimes, and say which periods those were.
  7. Remove the single best trade and see what happens to the statistics. A strategy whose edge vanishes when one outlier is deleted does not have an edge.
  8. Run it on a demo account for a minimum of one full session cycle and compare live fills against backtest fills. If you have real broker statements, use them; that is the only test that cannot be gamed.

The honest summary

Most EA backtests fail on statistics, not on logic. The rules may be fine. The problem is that a researcher who tried 300 configurations and reported the best one has manufactured a Sharpe ratio of about 1.5 out of nothing, and has no way to tell the difference from a real edge.

The defences are unglamorous and they work: count your trials, deflate your Sharpe ratio, hold out data properly, model your costs honestly, and check that the result survives its own best trade being removed.

None of this makes a strategy profitable. It makes the claim falsifiable, which is the only precondition for finding out.


Educational content only. Not investment advice. Past performance is not indicative of future results. Trading forex and CFDs carries substantial risk of loss.

Frequently asked questions

Why is a good backtest not evidence that a strategy works?

Because a backtest is a search process, not an observation. Once you have run enough variations, the best result is partly a function of the noise you happened to sample. The tester confirms what you already believed, which is why it feels like evidence.

What is multiple testing bias in trading?

It is the inflation of results that comes from testing many parameter sets and reporting only the best. If you test 100 variants and keep the top one, the reported performance is the maximum of 100 draws from the same noise distribution, not a representative outcome.

What is look-ahead bias?

Any use of information that would not have been available at the time the trade was taken — using the bar's close to decide an entry that the strategy places at the bar's open, or shifting a moving average by one bar in the wrong direction. It inflates results without any change to the underlying idea.

Why do understated costs matter so much?

High-frequency strategies pay their costs many times per trade, so a one-pip error repeated a thousand times can turn a losing system into an apparent winner. Costs must be modelled at realistic spread and commission, and the spread should be varied across the test.

What is the difference between a holdout set and a contaminated one?

A holdout is data you did not look at while developing the strategy. It becomes contaminated the moment you inspect it and then adjust anything, even slightly — at that point it is validation data, and you need fresh data for a true test.

What does an honest validation process look like?

Fix the parameters before testing on unseen data, use a test period that spans different regimes, report the drawdown and trade count rather than only the profit, and check whether the result survives realistic costs and reasonable parameter changes.

Sources

  1. Bailey, Borwein, López de Prado, Zhu — The Probability of Backtest Overfitting (2015) — accessed 2026-11-07
  2. Bailey & López de Prado — The Deflated Sharpe Ratio, Journal of Portfolio Management 40(5), 94-107 (2014) — accessed 2026-11-07
  3. Bailey & López de Prado — How Backtest Overfitting in Finance Leads to False Discoveries, Significance 18(6) (2021) — accessed 2026-11-07
  4. MetaTrader 5 — Strategy Optimization and Forward Testing — accessed 2026-11-07
  5. MQL5 — Testing Trading Strategies on Real Ticks — accessed 2026-11-07
  6. MQL5 — Creating and Testing Custom Symbols in MetaTrader 5 — accessed 2026-11-07
  7. Bailey & López de Prado — How to Spot Backtest Overfitting (slides) — accessed 2026-11-07

Use the free tools

Open-source calculators — no signup required.

Use the free tools
Risk disclosure — read before acting on anything above

All content on this site is provided for educational and informational purposes only. Nothing published here constitutes investment advice, a recommendation, or an offer to buy or sell any financial instrument. Trading foreign exchange, CFDs, and precious metals involves a high level of risk and may result in the loss of your entire capital. Past performance of any strategy, robot, or expert advisor (EA) — whether simulated, backtested, or live — is not indicative of future results. You are solely responsible for your trading decisions. If you are unsure whether a strategy is suitable for you, seek independent advice from a licensed financial professional in your jurisdiction.

Educational content only. Not investment advice. Trading forex and CFDs carries substantial risk of loss.