How Backtesting Works — And Why In-Sample Results Mislead

Backtesting is how you find out whether a trading idea has ever worked, without risking money to learn it. You take a rule — buy on this signal, exit on that one — and replay it over historical data to see what it would have done. Done honestly, it is one of the most valuable tools a systematic trader has. Done carelessly, it is a machine for generating beautiful, worthless equity curves.

The gap between those two outcomes is almost entirely about a handful of traps. If you understand them, you can read a backtest for what it is worth. If you do not, you will fool yourself — and the market will collect the tuition.

What a backtest actually does

At its core, a backtest is a simulation. You feed it historical prices, apply your entry and exit rules bar by bar, and record every trade it would have taken — wins, losses, drawdowns, the lot. Out comes a set of statistics: win rate, average win versus average loss, maximum drawdown, and often a risk-adjusted measure like the Sharpe ratio. Those numbers are your first, cheapest look at whether an idea has any merit.

The catch is that the simulation is only as honest as its assumptions. Real trading has costs, slippage and the awkward fact that you cannot see the future. A backtest that quietly ignores these produces a fantasy. The three biggest ways backtests lie are worth knowing by name.

Trap 1: Look-ahead bias

Look-ahead bias is using information in the test that you could not have had in real time. It is subtle and everywhere. Deciding to buy at the day's open using the day's closing price to pick the stock; using a revised data point that was not available until later; entering on a signal that was only confirmed after the bar closed but backdating the fill — all of these smuggle future knowledge into the past. The result is an equity curve that soars in the backtest and collapses live, because the live version never actually knows what the backtest secretly did.

Trap 2: Overfitting

Overfitting is the seductive one. You test a strategy, it is mediocre, so you tweak a parameter. Better. You add a filter. Better still. You keep adjusting — this indicator length, that time-of-day rule, this exception for Thursdays — until the backtest is gorgeous. What you have actually done is memorise the noise in your specific historical sample. The strategy now describes the past perfectly and predicts the future not at all.

The warning signs are a strategy with many finely tuned parameters, rules that seem arbitrary, and performance that falls apart if you nudge any setting slightly. A robust edge is stable — change the parameters a little and results degrade gracefully. An overfit one is a house of cards: it only stands in the exact configuration you tortured it into.

Trap 3: In-sample optimism

These two traps combine into a broader problem: any results measured on the same data you used to build and tune the strategy are optimistic by construction. Of course the strategy looks good on the data you shaped it around — you shaped it around that data. Numbers from this in-sample period tell you almost nothing about the future. The only results that matter are the ones from data the strategy has never seen.

The honest test: walk-forward

Walk-forward analysis is the discipline that keeps you honest. You split your history into an in-sample period, where you are allowed to build and optimise, and an out-of-sample period, which you hold back and never touch during development. You tune on the first, then test — once — on the second. If the edge survives on data it has never seen, you have real evidence. If it evaporates, you overfit.

1. Split the dataReserve a chunk of history the strategy will never see during development — the out-of-sample set.
2. Build in-sample onlyDesign, tune and optimise using the in-sample period alone. Pretend the held-out data does not exist.
3. Test once, out-of-sampleRun the finished strategy on the untouched data a single time. Repeatedly peeking and re-tuning turns it back into in-sample data.
4. Account for costs and slippageAdd realistic brokerage, taxes and slippage. A thin edge that survives in-sample often dies once real trading costs are charged.
5. Then paper tradeEven a clean walk-forward result is history. Run it live in paper mode before real capital to catch what the simulation missed.

How AIVITTA validates strategies

Every strategy in AIVITTA's algo trading platform is validated on roughly two years of NSE intraday data before it ships, with realistic assumptions about costs rather than a frictionless fantasy. Once live, every registered strategy runs continuously in paper mode by default — a real-time, out-of-sample shadow track — so you can watch it behave on data no backtest ever saw, with position sizing off a fixed risk percentage and a hard kill switch always within reach. The point is not to sell a backtest as proof; it is to keep testing the idea honestly, on live data, before a rupee is at stake.