Essays · The harness

Out-of-sample testing: the harness that kills most edges

By Daniel Roache · 14 July 2026 · 8 min read

This isn't about a strategy. It's about the harness we run every strategy through. Most ideas come out the other side with a headstone. That's not a bug; it's the entire point.

The default assumption: it's a mirage

We start every test from the same place: this edge is noise until proven otherwise. Base rates demand it. Search enough parameters across enough assets and some combination will look profitable by pure chance. So the backtest's job isn't to confirm the idea — it's to give the idea every honest chance to fail. Survive that, and you have something. Fail, and you saved the tuition.

Tier one: the Lab

The first tier is a fast simulation whose only job is to kill the large majority of ideas cheaply. It's approximate on purpose — enough fidelity to reject the obvious mirages, fast enough to run hundreds of parameter samples. Most ideas die here, and that's fine; they were never going to make it to the expensive test.

Tier two: the Courtroom

The rare survivors go to a full, fee-realistic, 1-minute, gap-aware replay. No dollar figure is trusted until it clears this tier. The honesty mechanics here are non-negotiable:

• Entries fill at the next bar's open. No look-ahead. The instant a fill happens on the same bar that generated the signal, you've booked profit from information you didn't have. This one bug flatters more strategies than any other.
• One position at a time. Clustered signals can't stack to inflate the statistics. Ten correlated "wins" count as one bet.
• A real t-statistic on independent trades — not a win rate. A win rate tells you nothing about whether the result could have come from a coin.
• Fees netted. A round-trip costs roughly 0.2R. Any edge smaller than the fee wall is structurally negative and just doesn't know it yet.

The part that does the killing: Leave-One-Regime-Out

Here is the piece most retail backtests skip. We keep a fixed library of distinct market regimes, chosen before the run: a blow-off top, bear markets, a bull, a range, a recovery. Then we hold out each regime once, fit the strategy on the rest, and score it on the regime it never saw. Rotate through all of them. A strategy passes only if it's profitable on a strong majority of the held-out regimes.

Why this matters: most "edges" are regime-conditional. They aren't a market truth — they're a description of one kind of market. Fit and test in the same regime and you'll never notice. This is where out-of-sample testing earns its keep, and it's a stricter cousin of walk-forward analysis: rather than only stepping forward in time, we deliberately withhold structurally different markets and demand the edge generalise to them.

The regimes are fixed in advance for a reason. If you pick the test window after seeing the result, you're not testing — you're cherry-picking a win and calling it science.

Why not just k-fold cross-validation?

Standard k-fold cross-validation shuffles rows at random, which for time-series data quietly mixes the future into the training set — look-ahead bias by construction. Any honest financial validation must respect the arrow of time: train on earlier data, test on later, and never let a fold peek forward. Leave-One-Regime-Out keeps that temporal integrity while adding the thing plain walk-forward can miss — an explicit demand that the edge work in market characters it was never shown.

The failure modes it exposes, over and over

• In-sample fitting — perfect on its training data, dead on anything else.
• Over-search — five hundred variants, one looks great; it's the 500-to-1 that fooled you.
• Autocorrelation — the "many trades" are one trade in costumes.
• Peak-not-plateau — profit sits on one exact parameter and craters beside it.
• Tail-carried profit — strip the top three trades and it goes flat.
• The fee wall — a real gross edge, smaller than the round-trip cost.

The takeaway

We publish the graveyard on purpose. The refutations earn the trust; the rare survivors are the product. An edge that clears next-bar fills, one position at a time, a real t-stat, netted fees, and profitability across held-out regimes it never saw — that's rare, and it's the only kind worth funding.

Next time someone shows you a curve, ask one question: which regimes did it never see, and did it still work? If they can't answer, you already have your answer.

Keep reading