One Out-of-Sample Test Is a Draw, Not a Verdict
Published 2026-07-26 · By Drew Shelem
Why a good backtest on held-out data can still fool you — and how to check
Article 6 of the cycle. In the previous articles we removed traps one at a time: the win rate that isn’t an edge; the Sharpe that measures the fee; the “best of a hundred runs” that isn’t a result; the costs a backtest never pays. Each article closed one way to fool yourself. This one closes the trap that feels safest of all: “but I tested it on data the strategy never saw.”
Where this topic even comes from
Picture an ordinary situation. You found a strategy on 5-minute BTC contracts. You did everything right: no peeking into the future, you split the history in two, tuned the parameters on the first half, and checked on the second half — the part the strategy “never saw.” On that held-out half the Sharpe came out to 1.5. (Sharpe, briefly, is return per unit of risk: the higher it is, the smoother and more reliable the profit. One-and-a-half is respectable.)
The question: do you go live?
Intuition says yes — you honestly checked on unseen data, and that’s the main defense against fooling yourself. That’s exactly the intuition this article breaks. Because that Sharpe of 1.5 is not an estimate of the strategy’s quality. It’s one draw. Shift the line between “tune” and “check” by a week and the number changes. It could land on 1.5. It could land on −0.8. You saw one value out of a wide spread and took it for the truth.
Below I’ll show this in numbers, explain how to get the whole spread instead of a single value, how to measure the probability that your best backtest is just luck — and, the twist, why one famous safety procedure isn’t needed at all on Polymarket’s markets, and you can drop it.
The thesis — in one paragraph, no jargon
A check on held-out data gives you one number, and that number is random: get lucky with how you sliced the history and it’s pretty, unlucky and it’s ugly. To tell a real edge from a lucky slice, you have to look not at one number but at the spread — run many different slices and watch how the result wanders. Out of that spread comes a very practical thing: how often the strategy that looked best during tuning fails on the check. If it fails often — around half the time — you optimized noise. And separately: the special procedure that scrubs “future leaking into the past” is only needed when trades overlap in time; on Polymarket, where every contract opens and closes inside its own window, there’s no overlap — so there’s nothing to scrub.
How you’d know I’m wrong. I ran two honest attempts to kill my own thesis; both failed in my favor, and I’ll show both on data. If, at a deliberately zero edge, the “best-in-tuning” strategy passed the check no better than a coin, the whole exercise would be empty. It doesn’t pass. And if scrubbing changed anything on non-overlapping trades, my conclusion “you don’t need it on Polymarket” would be a lie. It changes nothing.
All the numbers below are from synthetic data where the edge is deliberately zero. That’s on purpose: it lets you see the pure mechanism of self-deception, uncluttered by the question “was there an edge at all.” On your own data you’ll get your own numbers with the same script — it’s attached.
Let’s break down where the deception comes from
Step 1. Why one out-of-sample number is a coin toss
Take a fair coin and flip it ten times. It’s easy to get seven heads. Would you call the coin “lucky”? No — you know that’s spread, and you’d flip more. A backtest works by exactly the same logic, but somehow the logic gets forgotten. One check on a held-out chunk is one “session of ten flips.” It can produce a pretty number by pure chance — especially if you first tried many variants of the strategy and kept the best.
Here it is in numbers. I took 200 variants of one strategy, and all of them have exactly zero edge — pure noise, no signal inside. Then I did what everyone does: I picked the best variant on the tuning part.
| What we look at | Value | What it means in plain terms |
|---|---|---|
| Sharpe of the best variant during tuning | 2.49 | looks magnificent — “found the grail” |
| the same variant’s Sharpe on the check | −0.11 | on unseen data — zero-to-negative |
Look at that pair: 2.49 → −0.11. The strategy did not break between tuning and checking. It’s just that when you sift through 200 dud variants, the best of them isn’t the good one — it’s the luckiest one. Its pretty tuning Sharpe is a property of luck, not signal. On the check the luck doesn’t carry over, and zero is what’s left. That’s overfitting: you fit yourself to random patterns in the data and mistook them for a rule.
Step 2. How to see the whole spread instead of one point
Since one number is a random draw, it’s only sensible to draw many times and look at the whole spread. That’s exactly what a combinatorial check does (the full name is combinatorial cross-validation, but the name doesn’t matter here). The idea is simple, no formulas:
- Cut the history not into two parts, but into many chunks — say, 12.
- Then go through the options: declare some 6 chunks the “check” set, the other 6 the “tuning” set. And do that in every possible way to split 12 into 6+6.
- Each such way gives its own result on the check. There are many ways — 924. So instead of one number we get 924 different out-of-sample results on the very same data.
An analogy: instead of one exam, you run the strategy through 924 different exams built from the same material, and you look not at one grade but at the whole distribution of grades. One accidentally easy version of the exam won’t fool you — you see the whole spread.
Here’s what came out on those same zero-edge duds:
| What we look at | Value | What it means |
|---|---|---|
| Number of check-slices | 924 | instead of one point — a whole fan |
| Median Sharpe of the winner on the check | −0.46 | the center of the fan is below zero — as it should be at zero edge |
| Spread (5th…95th percentile) | [−2.17, +0.70] | that’s the width a single check squeezes into one point |
| Share of slices where the Sharpe came out positive | 0.31 | in a third of the slices, “zero edge” looks like a working strategy |
Now go back to the start, to your Sharpe of 1.5. It’s one point out of a fan like this. If your single check had handed you a value from the right tail, you’d go live with a dud in good conscience. The fan shows what a single check never shows: how much your number could have wandered.
Step 3. One number that gives a diagnosis — the “probability of overfitting”
From this fan you can compute a very practical quantity. In English it’s called PBO (probability of backtest overfitting); in plain terms it’s simpler: how often the variant that looked best during tuning turns out worse than average on the check.
The logic is direct. If the strategy has a real edge, the best-in-tuning variant will also be good on the check — it carries over. If there’s no edge, “best” is merely lucky, and on a new slice it’s equally likely to land above or below average. So at zero edge that share should be about half — like a coin toss.
On our duds it came out to 0.51. A coin, exactly. Translated to plain language: “your best backtest is no more distinguishable from chance than heads is from tails.” The closer this number is to zero, the more honestly the result carries over. The closer to 0.5, the more likely you optimized noise.
Why bother, if article 3 already had the Deflated Sharpe? Because they cure different diseases, and you need both. The Deflated Sharpe fixes one number: you tried a hundred variants — raise the significance bar, or the best Sharpe is inflated by the search. The “probability of overfitting” answers a different question — not “how high to raise the bar,” but “did the thing you picked carry over at all.” One is about the strictness of the threshold, the other about the fact of transfer. Different defects, different tools.
Step 4. About the famous “scrub” — and why on Polymarket it’s redundant
Books on this method always pair it with a companion — the purge procedure (plus its variant, embargo). Here’s what it is. Sometimes trades overlap in time: one trade’s label stretches several minutes forward and overlaps its neighbor. Then, when you slice, a chunk “from the future” can leak into the tuning part, and the strategy is effectively peeking at the answers. Purge means scrubbing out of the tuning part the trades that overlap with the check part. Analogy: don’t let a student practice on homework whose problems coincide with tomorrow’s exam.
How much this matters is easy to show. I deliberately made overlapping labels (each stretches 20 steps forward, neighbors nearly identical) and sliced naively:
| Scheme | How much the “prediction” matched the fact | Sharpe | What happened |
|---|---|---|---|
| Naive slicing, labels overlap | very strongly (0.95) | +84 | the strategy “peeked” at neighbors — Sharpe 84 from pure noise |
| With scrubbing (purge), same data | not at all (0.00) | +0.2 | scrubbing removed the leak — back to zero |
| Control: labels do not overlap | — | −0.2 | no leak — nothing to scrub |
Look at the top row: overlapping labels birthed a Sharpe of 84 out of thin air. That’s what scrubbing is for — where it’s needed. Now the bottom row, and this is the key one for us. On Polymarket a trade opens and closes inside a single contract window and doesn’t stretch into the next — you confirmed this yourself; otherwise these markets can’t work. So the labels of neighboring contracts don’t overlap. This is exactly the control case: no leak, nothing to scrub. Dragging purge in “just in case,” copied from someone else’s pipeline, is a ritual with no content. Before you copy a defense, check what it defends against, and whether you have that disease at all.
What we take to our market from all this
- The fan of paths and the “probability of overfitting” — we take. Replace one out-of-sample number with a distribution and with the diagnosis “how often the winner fails.” It works on these markets the same way it does on synthetic data.
- The scrub (purge/embargo) — by default, we don’t. Trades don’t overlap between windows — there’s nothing to treat.
What would refute this — and what the method does NOT give
This is the most important section: where I could be lying, and where the method is powerless.
Two conditions on which the thesis would collapse (and didn’t):
- If, at zero edge, the “probability of overfitting” did not pull toward half, the diagnosis would be useless. It came out 0.51.
- If scrubbing changed anything on non-overlapping trades, the conclusion “you don’t need it on Polymarket” would be wrong. It doesn’t: with overlap, Sharpe 84 versus zero; without overlap, zero in both cases.
What the fan does NOT give, and this matters more than it seems. The 924 slices are built on the same data, just reshuffled. More slices is not more information — it’s reuse of the same history. And there’s a subtlety the method doesn’t fix: consecutive 5-minute windows are not independent observations. One volatility regime stretches across several contracts in a row, the way bad weather holds for several hours. So the slices are correlated with each other, and the true spread on live data is wider than the fan shown here, [−2.17, +0.70]. Purge doesn’t help with this — it’s about label overlap, not about the process itself being inertial. That separate disease — serial correlation — is the subject of the next article in the cycle; here I name it honestly rather than sweep it away.
What I don’t know, and where the limits are
- All the numbers are from synthetic data. The edge on them is deliberately zero. This is a demonstration of the mechanism (the diagnosis pulls toward 0.5; the leak appears from overlap), not a measurement of your strategy. On your own data, run the script.
- The market’s inertia (serial correlation) is not removed by scrubbing or by the fan. It adds confidence that isn’t there — the same error as “treating all trades as independent.” That’s the bridge to the next article.
- Within a single window several entries in a row can overlap each other — a smaller effect than the between-window one; I didn’t measure it here, but with frequent intra-window trading it’s worth checking.
- The combinatorial slicing scrambles the order of time — training can land on “later” chunks than the check. For pure forecasting tasks that’s a legitimate objection; on a stable stretch it’s tolerable, on a shifting one it’s not.
- The “probability of overfitting” is a probability, not proof of an edge. A low value removes one way to fool yourself. It does not say an edge exists.
What to do tomorrow
- Don’t trust one out-of-sample number. It’s a point from a fan. Run many slices and look at the median and the spread. If the median is near zero and “your” number is from the right tail — you don’t have a strategy, you have a lucky slice.
- Read the “probability of overfitting” as a diagnosis. Around 0.5 — your best variant is indistinguishable from luck, you optimized noise. Closer to zero — it carries over. This isn’t about the significance threshold, it’s about the fact of transfer.
- Don’t copy purge if your trades don’t overlap. Check your label’s horizon. On Polymarket a trade lives inside a window → there’s no overlap between windows → scrubbing isn’t needed. A redundant check is a defect too, one dressed up as diligence.
- Account for the market’s inertia separately. The fan doesn’t catch it, so don’t mistake its narrowness for confidence: on correlated windows the real spread is wider. For now — at least know this.
- Don’t drop the trial count from article 3. The Deflated Sharpe and the “probability of overfitting” catch different things; you need both. You can run the fan on your own trade log without risking money, in paper mode — free, before any subscription: the signal engine there is the same one used in live trading, so the fan is built on real execution, not an idealized one.
FAQ
If I honestly checked on held-out data, isn’t that enough? It isn’t. One check is one draw. A pretty number could have landed by luck, especially after a variant search. You need the spread across many slices, not one point.
Do I need purge on Polymarket? Most often no. Purge scrubs the leak from trades that overlap in time. If a trade opens and closes inside a contract window, there’s no overlap between windows — nothing to scrub. First check your label’s horizon, then decide.
Does this combinatorial check replace ordinary walk-forward? It complements it. Walk-forward gives one path through time and one number; the combinatorial one gives a fan and a diagnosis. One number doesn’t show its own spread; the fan does.
Is a “probability of overfitting” of 0.5 bad? It’s the worst case. It means the best-in-tuning variant fails the check in half the slices — exactly like a coin at zero edge. Closer to zero is better.
More slices means more data, so more reliable? No. The slices are built on the same data, just reshuffled. It’s an honest reworking of what you have, not new information. And because of the market’s inertia the slices aren’t even independent, so the fan actually understates the real spread.
Does all this prove I have an edge? No. It removes one way to fool yourself — mistaking a lucky slice for quality. You can’t prove an edge this way; you only prove that a particular winner doesn’t fall apart on transfer.
Disclaimer
This material is educational and is not financial advice. Past results do not predict future results. Trading on prediction markets carries real risk, up to and including the total loss of capital. Access to Polymarket is restricted or prohibited in some jurisdictions — verify legality where you live before trading.