ALGO TRADING
Why a strategy that worked in simulation underperforms with real capital, how to tell the three causes apart, and how to prove the live side is real.
– THE GAP
A backtest is a simulation of a strategy over historical data. A live track record is what the same strategy actually did with real capital. When the two disagree, the useful question is not whether the backtest lied. It is which of three distinct mechanisms produced the gap, because each one has a different diagnostic and a different fix.
Some degradation is expected and carries no information. An out-of-sample result drawn from the same distribution sits below an in-sample optimum on average, for the simple reason that the optimum was selected out of many candidates. The rest of the gap is not statistical at all: it points at a cost model that was too generous, a leak in the signal, or an execution problem that never appeared in the simulation.
Making the comparison at all requires both sides to be measurable. A backtest that applies no fees is not comparable to anything. A live record made of screenshots is not comparable either, and it is certainly not comparable for anyone who was not the one taking the screenshots.
– THREE CAUSES
Most post-mortems collapse these into one vague conclusion about overfitting. They are separable, and separating them is what tells you whether the strategy is salvageable.
Every parameter value you tried, every rule you added and removed, every window you shifted, counts as a trial. The best of N trials is biased upward even when none of the variants holds a genuine edge. This is what the Probability of Backtest Overfitting and the Deflated Sharpe Ratio (Bailey and Lopez de Prado) were designed to quantify.
Count the true number of configurations evaluated, almost always several times the number you remember, then compute the PBO. If the deflated Sharpe falls to zero, there was never a demonstrated edge to lose.
Fees, slippage, funding, borrow costs, partial fills and the spread all subtract from a live account and cost nothing in a naive simulation. Look-ahead leakage is the harsher version of the same problem: a signal computed on a bar that had not closed yet is free money in a backtest and simply unavailable live.
Re-run the same backtest with realistic per-venue fees and a slippage assumption. If the edge evaporates at 5 basis points, the edge was the cost model, not the strategy.
Downtime, a manual override during a drawdown, a position capped by available margin, an order the venue rejected, a data feed that dropped for an afternoon. Each of these breaks the equivalence between the two series without either series being wrong on its own terms.
Compare exposure over time rather than returns. If the live account sat flat on days the simulation was in the market, the gap is operational and no amount of re-fitting will close it.
– THE BACKTEST SIDE
Before comparing anything, the simulated side has to be capable of losing money the way a real account loses money. This is the minimum for the comparison to carry information.
Not every backtesting engine models all of it, and the ones that skip execution realism are the ones that produce the widest gaps. If you want a stack that covers the list, ManifoldBT is a Python backtesting library with a Rust core that runs sequential fills with fees, slippage and funding, alongside walk-forward, Monte Carlo and look-ahead detection.
Disclosure: ManifoldBT is built by the same team as AuditZK. It is source-available under Apache 2.0 with Commons Clause; walk-forward and the safety checks sit in its Pro tier.
– THE LIVE SIDE
The backtest side gets all the attention, and then the live side is a spreadsheet the trader assembled. That asymmetry is why most backtest versus live comparisons convince nobody.
The live series has to be continuous. Daily portfolio valuations, not a list of closed trades, because unrealized losses live in the space between the trades.
It has to be time-weighted rather than raw P&L, otherwise every deposit and withdrawal reads as performance and the comparison is measuring your funding schedule.
It has to come from the venue rather than from the trader. A comparison you ran on your own numbers proves something to you and nothing to a third party.
It has to start when you say it started. A record assembled after a good quarter carries backfill bias, which is the live-side twin of backtest selection.
– HOW TO COMPARE
Six steps, in order. Skipping any of the first four produces a number that looks like a comparison without being one.
| Step | What to do | Why it matters |
|---|---|---|
| Pick the windows | Compare live against the out-of-sample segment of the backtest, never the in-sample one | The in-sample segment is where the parameters were fitted, so it was never a prediction |
| Normalize the metric | Time-weighted return on both sides, same risk-free rate, same annualization convention | A Sharpe computed two different ways is not a comparison, it is a coincidence |
| Bound the noise | Establish how many live observations you have before reading any Sharpe difference as real | A short live sample cannot distinguish a 1.2 from a 0.6 at any useful confidence |
| Deflate the backtest | Apply the PBO and the deflated Sharpe using the true number of trials | Reduces the backtest figure to what was demonstrated rather than what was selected |
| Overlay exposure | Plot time in market for both series, not only returns | Separates an operational gap from a statistical one, which nothing else does |
| Publish the live side | Use a verified record so the comparison is checkable by someone who does not trust you | Otherwise the whole exercise is another self-reported claim |
FAQ
Related Reading
NEXT STEP
Connect your accounts read-only. AuditZK builds the verified live record your backtest gets compared against, without exposing individual trades or strategy logic.