Monte Carlo Only Tells You Whether the Set of Trades Flattered You
Published 2026-07-28 · By Drew Shelem
Why a green “robust” verdict is not proof of an edge, and three things this test doesn’t see
Article 9 of the cycle. You ran the backtest, opened the validation tab, and saw: “Monte Carlo robustness: Fragility 8% — robust.” A green check, a feeling that the strategy is validated. This article is about how that green check checked far less than you think — and exactly where, in that gap, you fool yourself.
Where this topic even comes from
A Monte Carlo check sits in almost every backtester, and traders treat its verdict as a final stamp of “the strategy is sound.” But it answers one narrow question: did the set of trades that landed in your sample flatter you. That’s an honest and useful question — but it’s not the one you hear. You read “robust” as “the edge is real.” Those aren’t the same thing, and the whole article is about how much they aren’t.
Let’s work through what this check actually does, the one question it does close — and the three big questions it doesn’t touch at all, even though those are the ones you wanted closed.
What this is about — briefly
The Monte Carlo bootstrap takes your trades, reassembles random “alternative histories of the same strategy” out of them many times over, and looks at what fraction of those histories still ended in profit. That fraction-in-the-red is called Fragility; a low Fragility earns the verdict “robust.”
The useful thing it tells you is exactly one: whether your profit rested on a lucky set of trades. That’s a real check, and I’m not here to bury it.
And here are three things it does not check — the three you most likely take for checked:
- Overfitting. An overfit dud passes Monte Carlo as “robust.” I’ll show it on a run.
- Transfer to new data. Reassembling your own trades is an in-sample test; about data the strategy hasn’t seen, it says nothing.
- Independence of trades. The standard bootstrap reassembles trades one at a time, which breaks the runs and thereby assumes trades are independent; on clumped trades that makes the verdict rosier than the truth.
All the numbers come from synthetic data, where I set the edge myself so the pure mechanism is visible.
Let’s break it down step by step
Step 1. What the Monte Carlo check actually does
First, how it’s built, without formulas. You have a list of backtest trades with their profits and losses. Picture them as a bag of chips, each chip carrying the result of one trade. The check draws a chip, records its result, and puts it back in the bag; draws the next, and puts it back again; and does this as many times as you had trades. Because each chip goes back, on the next draw it’s available again — so the same trade can come up twice or three times, while another might not come up at all. The result is a sample of the same size, but it’s no longer your original list; it’s a random reassembly of it: a different set of trades, in some of them more of the lucky ones, in others fewer. A thousand such reassemblies get built.
Why “with replacement” rather than just a shuffle? If you drew without replacement, you’d get exactly the same trades in a different order — and the final sum of profits and losses doesn’t depend on order, so all thousand “alternative histories” would give the same result, and there’d be nothing to check. Replacement is precisely what creates different sets — and only that produces a spread of outcomes.
For each of the thousand reassemblies you compute the final result, and look at the fraction of them that ended in the red. That fraction is Fragility. A low one means your profit rested not on one particular lucky set of trades, but is robust to which of them happened to land in the sample.
There’s also a picture — the fan chart: over a gray cloud of alternative curves, your real run is drawn as a bold line. If the bold line runs noticeably above the gray cloud, your real result came out luckier than a typical reassembly — that is, you got lucky with the particular set of trades that actually happened.
So the whole honest meaning of the check is this: it tells you whether the set of trades flattered you. That’s useful — but narrow. Notice: it works with the set of trades you already have, and doesn’t for a second question that set itself. That’s where all three of its blind spots come from.
Step 2. It doesn’t catch overfitting — and that’s the big one
Here’s the most dangerous gap. Let’s test it directly: take 200 variants of a strategy, all with a true edge of exactly zero — pure noise, no signal. And do what everyone who tunes parameters does: keep the best variant by the backtest’s final result. That’s the overfitting from article 3 — the best of the sifted duds looks profitable purely by the luck of the selection.
Now run that very Monte Carlo check on this “winner”:
| What we look at | Value | What it means |
|---|---|---|
| Backtest result of the best of 200 | a solid profit | but that’s selection luck, not an edge |
| Monte Carlo Fragility | 0.8% | verdict “robust” |
| Result on fresh data | about zero | the true edge really is zero |
Stop here. The check called “robust” a strategy that has no edge whatsoever. Why did that happen? Because the bootstrap reassembles the same lucky trades. Since the original set came out in profit, almost any reassembly of it is also in profit — hence the low Fragility. The check asks “is the profit robust to reassembling this set of trades,” not “was that set of trades itself a lucky fluke of the selection.” The second question — the one about overfitting — it never asks. So a green verdict on an overfit dud isn’t a malfunction, it’s exactly what you should expect from it. “Robust” does not mean “real.”
Step 3. It doesn’t test transfer to new data
Reassembling your trades is a game with one and the same set of data, shuffled back and forth. It’s in-sample by definition. About data the strategy hasn’t seen, it can say nothing — there’s nothing there to reassemble. And it’s precisely transfer to new data that separates a real edge from a fit; for that you need the out-of-sample fan from article 6, not a bootstrap. These are two different questions. You passed Monte Carlo — you learned nothing about whether the edge survives on fresh data.
Step 4. It assumes your trades are independent — and they’re not
The last gap is subtle but important, and it comes straight from article 7. The standard bootstrap reassembles trades one at a time. And because it shuffles them individually, it tears the runs apart — and thereby treats trades as independent. But losses, as we saw, clump. On clumped trades, reassembling one at a time gives a spread narrower than the truth, which means Fragility comes out rosier than it really is.
Let’s check. Take an inertial strategy and average over many of its runs, so as not to depend on one lucky one:
| Way of reassembling | Median Fragility | What it assumes |
|---|---|---|
| One trade at a time (as in the product) | 34% | trades are independent — false |
| In chunks, keeping the runs | 40% | runs intact — honest |
The gap is modest on average, but systematic: reassembling one at a time is always rosier. And here’s what matters more than the average — in 22% of the runs the honest, block reassembly gives a more worrying verdict than the per-trade one: where one-at-a-time shows you “acceptable,” in chunks it comes out “fragile.” That is, in more than one run in five the standard Monte Carlo hands you a calmer estimate than the truth.
From this comes a practical conclusion: the autocorrelation detector and the way of reassembling have to be wired together. If a validation tool separately detects serial correlation (such a test usually sits right there), then at significant autocorrelation its Monte Carlo should switch from per-trade reassembly to block reassembly — otherwise the “robust” verdict stays falsely rosy precisely on inertial strategies. When the detector and the reassembly are wired together, that hole is closed: the correlation test itself turns on the honest mode.
What would refute this — and where the limits are
Let me say up front, so I don’t overreach: the Monte Carlo check is not useless. On its narrow question — “did my profit rest on a lucky set of trades within my sample” — it answers honestly, and the fan chart is a good check against self-deception. The article’s thesis isn’t “throw out Monte Carlo,” it’s “it answers one narrow question; don’t read it as the whole validation.”
The condition on which the thesis would collapse (and didn’t): if an overfit dud did not pass Monte Carlo as robust, the check would be catching overfitting, and my main criticism would be wrong. But it passes, with a Fragility of 0.8%. And if per-trade reassembly weren’t systematically rosier than block reassembly on clumped trades, the independence criticism would be empty. But it is rosier, and in one run in five it changes the verdict.
What I don’t know, and where the limits are
- All the numbers are from synthetic data. I set the edge myself. This is a demonstration of the mechanism (the dud passes; per-trade is more optimistic), not a measurement of your strategy.
- The chunk length in block reassembly is your choice, and it affects the result. Take it on the order of a typical run length; at a different length the numbers shift.
- Monte Carlo’s narrow question is legitimate. I’m not abolishing the check, I’m putting it in its place: it’s about luck in the set of trades, not about the edge.
- The verdict thresholds (robust / acceptable / fragile) are a convention, not a law of nature; treat them as rough guides.
What to do tomorrow
- Read Monte Carlo literally: “did the set of trades flatter me.” Not “is my edge real.” A green “robust” closes only the question of luck in the sample’s composition — which particular trades landed in it.
- Don’t let the green check stop the harder tests. Monte Carlo replaces neither trial counting (article 3), nor the out-of-sample fan (article 6), nor the sample-based confidence interval (article 1). It goes alongside them, not instead of them.
- At autocorrelation, the reassembly should become block-based. The per-trade bootstrap understates the risk on clumped trades; in chunks it’s honest. A good tool does this itself: in gogobots the Monte Carlo switches to block reassembly automatically when the autocorrelation test (Ljung-Box) fires — you don’t have to remember it.
- Read the fan chart correctly. Your real line high above the gray cloud is not “well done me,” it’s “I got lucky with the set of trades”; exactly what the check answers, and no more.
- Ask about transfer separately. The only way to learn whether the edge survives on new data is new data. You can gather an honest sample outside the backtest, without risking money, in paper mode — free, before any subscription: the same signal engine as in live trading.
FAQ
My Fragility is 8%, the verdict is robust — does that mean the edge is real? No. At worst it means your profit didn’t rest on a lucky set of trades within the sample. An overfit dud with zero edge passes this check just as easily. “Robust” is not “real.”
Then what is Monte Carlo good for at all? It has a real, narrow use: to show whether the set of trades in your sample flattered you, and through the fan chart, whether your run came out luckier than a typical reassembly. That’s a defense against one kind of self-deception, not against all of them.
Does Monte Carlo replace the out-of-sample check? No, they’re different questions. The bootstrap shuffles your own trades (in-sample); transfer is about new data. You passed Monte Carlo — you learned nothing about transfer.
Why does reassembling one trade at a time understate the risk? Because it tears the runs apart and pretends the trades are independent. On clumped trades the true spread is wider, and the per-trade bootstrap underestimates it — Fragility comes out rosier than the truth.
What do I do if the test shows autocorrelation? Switch Monte Carlo to block reassembly (in consecutive chunks, not one at a time). Otherwise the “robust” verdict is too optimistic exactly where inertia is strong. A good tool makes this switch itself, when the autocorrelation test fires.
Disclaimer
This material is educational and is not financial advice. Past results do not predict future results. Trading on prediction markets carries real risk, up to and including the total loss of capital. Access to Polymarket is restricted or prohibited in some jurisdictions — verify legality where you live before trading.