Backtesting Without Self-Deception: How to Rigorously Test a Strategy

A historical strategy test is like a rehearsal before going live. The problem is that most traders unknowingly "cheat" — and then wonder why live trading diverges so dramatically from the simulation. This article breaks down the six main backtesting traps and shows how to avoid them using rigorous methodology.

---

1. Survivorship Bias – Survivors Lie

Imagine testing a strategy on the current constituents of the S&P 500. Your dataset automatically includes only companies that survived to the present day. Companies like Enron, Lehman Brothers, or Bed Bath & Beyond — all removed from the index — are missing from your test.

Result: The strategy appears better than it actually was, because it only works with "winners."

Solution: Use point-in-time data — a database that captures the exact composition of an index at a specific historical date, including companies that were members at the time but later went bankrupt or were delisted.

---

2. Look-Ahead Bias – Knowledge from the Future

This trap is more insidious. It occurs when a model uses information at time T that was only actually available at time T+1 or later.

Examples:

• Earnings results published February 15 are used to generate a signal on February 1.
• An annual dividend announced in Q4 is incorporated into a Q3 signal.
• Using the "day's closing price" for an intraday signal that should have been generated before the close.

Result: The simulated strategy "knows" things that a real trader couldn't possibly have known at that moment — performance is fiction.

Solution: Every data point must carry a correctly labeled availability date (reporting date), not the date the underlying event occurred.

---

3. Overfitting / Curve-Fitting – Tailoring to Dead Data

This is arguably the most widespread trap. Consider a strategy with 12 parameters (moving average length, RSI band, stop-loss level…). If these parameters are optimized on a single historical dataset, the strategy "learns" that specific data — including its random fluctuations.

Analogy: It's like writing an exam and then giving a student that exact exam to memorize. They'll ace it perfectly — but won't be able to answer any different questions.

How to spot overfitting:

Symptom	Indicator of Over-Optimization
Sharpe ratio > 3 in backtest	Suspicious — real strategies achieve 0.5–1.5
Maximum drawdown < 5%	Nearly impossible over a longer horizon
Parameters >> number of trades	Too many degrees of freedom
Performance drops >50% in OOS test	Strong signal of overfitting

---

4. Ignoring Fees and Slippage

A 0.1% brokerage fee per side looks negligible. But a strategy trading 200 times per year pays 40% of capital in fees alone (200 × 0.2% round-trip).

Slippage is the difference between the price assumed in the backtest (typically close/open) and the price actually achieved when an order is filled — especially for less liquid securities or large positions.

A strategy on a 100,000 CZK portfolio with 300% annual turnover:

• Fees: 300,000 CZK × 0.2% = 600 CZK/year (small account, relatively manageable)
• But on an institutional account of 10M CZK with the same turnover: 60,000 CZK annually — that's an additional 0.6% p.a. the backtest never accounted for

For short strategies, borrow cost (the cost to borrow shares) adds another layer that backtests almost never include.

---

5. Sample Size Too Small

A strategy generated 23 trades over 2 years and produced a Sharpe of 1.8. Impressive? Statistically: with 23 trades, the confidence interval is extremely wide — the result could be pure chance.

Basic rule: For statistically meaningful conclusions, you need at minimum 100–200 closed trades and ideally coverage of multiple distinct market regimes (bull, bear, consolidation, high/low volatility).

Data horizon and the shallowest factor: A test is only as deep as the data availability of its least-covered factor. If one factor (e.g., sentiment data, alternative data) only exists from 2018, the entire backtest is effectively limited to that horizon — even if price data goes back to 1990.

---

6. In-Sample vs. Out-of-Sample: Train/Test Split

This is the foundational methodological principle of any serious backtesting process.

Train/Test Split

Data is divided into:

• In-sample (IS) / train set – this is where parameters are discovered and optimized
• Out-of-sample (OOS) / test set – this is where the strategy is validated only, with no further optimization

Why do in-sample results overestimate?
The optimizer will always find a configuration that works on training data. Part of that "performance" is adaptation to random noise in the data, not to a genuine market pattern. OOS testing reveals how much of the performance transfers to unseen data.

Practical guidance:

• Typical split: 70% IS, 30% OOS
• OOS must always be chronologically later — not randomly selected

Walk-Forward Validation

A more sophisticated approach. Instead of a one-time split, we perform rolling optimization:

[IS₁ → OOS₁] → [IS₂ → OOS₂] → [IS₃ → OOS₃] → ...

In each window, we optimize on IS data and collect results only from the OOS window. The resulting equity curve is composed exclusively of OOS segments — this is the most rigorous backtest available.

Advantage: Walk-forward reveals whether a strategy is robust across different market conditions, or whether it only works in one specific period.

---

How to Do This in QMA

QMA's Strategy Lab at /strategy was designed with the pitfalls described above in mind. Here's how to use it methodically:

The ❌ Over-Optimized Strategy Label:
If the algorithm detects signs of curve-fitting — such as an excessively high ratio of parameters to trades, or a sharp performance drop under small parameter changes (parameter sensitivity) — the strategy receives a visible ❌ label in the interface. This label does not mean the strategy is unusable; it signals that results must be interpreted with significant caution and that OOS validation is mandatory.

In-Sample vs. Out-of-Sample Panel:
In the /strategy section, you can define the date range for the IS and OOS segments. The platform automatically displays key metrics (Sharpe, Max Drawdown, Win Rate) separately for each segment, making it immediately visible how much performance degrades in the transition from IS to OOS.

Walk-Forward Engine:
For advanced users, a walk-forward module is available that automatically performs rolling optimization and assembles an OOS equity curve. Results are accompanied by a distribution of Sharpe ratios across individual OOS windows — a robust strategy should show consistent values without extreme outliers.

Costs and Slippage:
When defining a strategy, you can input a brokerage fee (percentage or fixed) and estimated slippage in basis points. The platform recalculates performance net of costs — a critical difference from naive backtesting.

Data Depth and Factor Availability:
On the /stocks/[symbol] page and within the strategy lab, each factor displays its data availability. If a strategy uses a factor only available from 2019, the walk-forward test automatically limits the horizon to that date.

Portfolio Context:
Strategy results can be linked to /portfolio, where it becomes visible how the strategy would affect overall exposure and correlation with existing positions.

---

Summary: Rigorous Backtesting Checklist

Check	Why It Matters
✅ Point-in-time data	Eliminates survivorship bias
✅ Correct reporting dates	Eliminates look-ahead bias
✅ OOS / walk-forward validation	Exposes overfitting
✅ Fees + slippage in the model	Realistic performance
✅ Min. 100+ trades	Statistical relevance
✅ Coverage of multiple market regimes	Strategy robustness
✅ Data horizon = shallowest factor	Consistent test scope
✅ Parameter sensitivity analysis	Detects curve-fitting

Backtesting is a powerful tool — but only when conducted honestly. A strategy that survives rigorous OOS validation and still delivers an acceptable Sharpe ratio net of costs has a genuine chance of holding up in live trading.

---

Disclaimer

Past performance is not a guarantee of future results. All investment decisions are your own responsibility.