A backtest youare allowed todisbelieve
Most backtests are flattering for three well-known reasons, and this one is built to remove all three: it walks forward instead of fitting the whole history, it fails the run when a feature has read the future, and it prices trades with the costs the venue actually charged us.
The three ways a backtest lies
It has read the future
A feature computed on the whole series — a normalisation, a rolling statistic aligned one bar wrong, a label that peeks past its horizon — produces a curve that goes up and a strategy that does not. Every run here carries a leakage check, and it is not advisory: a run whose features could see past their own timestamp is reported as failed, not as a good result with a warning underneath.
It was fitted to the whole history
Choosing parameters on all the data and then testing on all the data measures memory, not skill. Runs are walk-forward: fit on a window, test on the window after it, step, repeat. The number you are shown is the out-of-sample one, and the in-sample one is shown beside it precisely so the gap between them is visible.
It priced trades at the published fee
One venue we trade publishes a zero-fee schedule and charged us fourteen basis points round trip. Another fills a large order several tenths of a percent away from where a naive test assumes. So costs come from what our own fills actually cost, per venue and per instrument size — a strategy with a thin edge lives or dies exactly here, and it should die in the test rather than in your account.
And a fourth: the data had holes
A dataset missing bars produces a strategy that never had to trade through the moments that were missing, which are rarely the calm ones. Runs report their coverage — how many rows were dropped and why — because a result computed on 87% of the period is a different result, and you should be told which one you are looking at.
From an idea to a number you can act on
Four steps, and a strategy cannot skip to the end: promotion to live money requires that a real backtest exists, passed, and is attached to the strategy.
- 01
Describe it
In plain language or as explicit conditions. The assistant turns it into a strategy definition you can read and edit — it is code and rules, not a black box you are asked to trust.
- 02
Run it
Walk-forward across the history the venue actually has, with costs, slippage and the leakage check. Long runs report progress and can be cancelled.
- 03
Read the whole report
Return, drawdown, hit rate, exposure, the distribution of outcomes rather than the average, and the gap between in-sample and out-of-sample. A single headline number is how a bad strategy gets promoted.
- 04
Paper, then live
A passing backtest earns paper trading, not real money. Live promotion is a separate decision you sign, and the risk gate applies to every order the strategy produces afterwards.
Questions about backtests
How far back does the history go?
Roughly two years on the crypto venues, deepening as we keep collecting. The run tells you the period it actually used, which matters more than the maximum: a strategy tested through one regime has been tested against one market’s weather.
Can I trust a good result?
Trust it less than you want to. Look at the out-of-sample gap, the drawdown, the number of trades, and whether the edge survives the costs. The system is built to make those visible; it cannot make them good.
Does the AI pick the parameters?
It can search a grid and propose, and the search is what it is — a machine trying combinations. A parameter set that only works in one window is exactly what walk-forward exists to expose, and the report shows you the windows.
What stops a strategy going live on a lucky run?
Promotion checks the strategy’s status and that a report is attached, and the step from paper to live money is a human decision recorded as one. Nothing is promoted by an automated process reading a good number.
Test the idea you keep meaning to test
Paper accounts run the full backtester against the same history and the same cost model. The only thing missing is the money.