The Best of N Is Not a Result

The Best of N Is Not a Result

The third article in the validation series. It follows The Sharpe Ratio Measures the Fee. Every number is reproducible — script linked at the end.

---

If you read nothing else: if you tried several variants of a strategy and kept the best one, its result is inflated — and the more variants you tried, the more inflated it is. So the number of variants tested has to be counted and published alongside the result. Without it the result cannot be assessed.

Start with what you do every day

You build a strategy. You pick a stretch of history. You run a backtest — the result is mediocre. You change the entry threshold and run again — better. You nudge a couple more parameters, do ten or fifteen runs, and find a combination with a respectable Sharpe ratio.

Then you do the careful thing: you take a different stretch of history, one that played no part in the tuning, and check the strategy on it. That is the right step, it is called out-of-sample testing, and plenty of people skip it.

The process looks conscientious. Every backtest was honest, the platform embellished nothing, and you even added an extra check.

And yet there is a hole in this process through which most “working” strategies drain away. It is not in the backtester and not in the data. It is in the step where you picked the best result out of fifteen.

---

The demonstration

To show this cleanly you need data with nothing in it to find. I took a random walk — a series where each value is the previous one plus a random number. It contains no pattern by construction: this is not a “difficult market,” it is a random number generator.

I sliced it into binary markets: “will the price be higher twenty steps from now,” quoted at exactly 50¢.

Then I built 451 strategies. Real ones, with parameters: “if the rise over the last L steps exceeded threshold θ, bet YES, otherwise NO.” Eleven values of L, forty-one values of θ, every combination.

What matters is what happens to these strategies:

• They cannot predict anything. There is no pattern in the series, so every one of the 451 is right exactly half the time.
• All of them pay the fee — 1.75pp per trade, as on a real Polymarket crypto market.
• Therefore the average result of every single one is negative, and that is not a guess but arithmetic: right half the time, paying a fee, losing money.

Then comes your workflow exactly. Split the data in half. On the first half, pick the best strategy by Sharpe. On the second half, which the selection never saw, check it.

The “t” column shows how far the result sits from zero in units of its own margin of error. The familiar significance threshold is two. More on this below.

Read the top row. Two hundred and fifty trades per configuration is realistic: a couple of months of fifteen-minute market history. A sweep of 451 variants produced a strategy with a positive Sharpe out of a process whose average result is guaranteed to be negative.

Not “noisy.” Not “weak.” Guaranteed to lose, because the fee is known and was subtracted in advance.

Now the right-hand column. It does not move: about −0.03 in every row. That is the truth, and it is the same regardless of how hard the search worked. Everything the selection found stayed inside the first half of the data and never came out.

And the third thing, which is what the whole table is for. Look down the first column: the more trades there are, the worse selection is at deceiving you. At ten thousand it can no longer drag the result into positive territory — there is not enough random fluctuation to cover the fee.

Sample size is not a textbook formality. It is what determines whether your defence against self-deception works at all.

---

The claim, stated so it can be broken

I claim: a Sharpe ratio obtained by picking the best of N variants is inflated, and the inflation grows with N. So a backtest result without a stated N is not interpretable — not “weakly interpretable,” but not interpretable at all.

I am wrong if: the expected best Sharpe among N deliberately empty strategies does not grow with N. That is one statement, it has a formula, and it can be checked two ways in the attached script — by calculation and by simulation.

---

Why this happens

Nothing here is specific to trading, and it can be understood without formulas.

Imagine you flip twenty coins a hundred times each. One of them — say the seventh — comes up heads 62 times. You compute the probability of a deviation that large for a fair coin: it is small, around 1.7%. Conclusion: the seventh coin is loaded?

No. The calculation is correct for one coin chosen in advance. But you chose the seventh after looking at all twenty. Among twenty fair coins, one with a large deviation is close to certain — and that is the one you found.

Backtests work the same way. Every run is honest. The error enters at the moment you look at all the results, take the best, and then assess it as though it had been the only one.

Formally: each run’s result is the strategy’s true quality plus a random deviation. The maximum of many such numbers contains the largest random deviation that happened to occur. The more runs, the larger the deviation you find — and the more of your “result” is actually luck.

How much is it inflated by

The good news is that the size of the inflation is not a matter of guesswork; it has been calculated. Bailey and López de Prado derived a formula for the expected best Sharpe among N strategies each of which has a true quality of exactly zero.

Read the table as: “if I sweep this many variants of deliberately empty strategies, the best of them will show this Sharpe — purely by chance.”

A thousand variants on pure noise gives an expected Sharpe of 3.26 — a number that gets money placed under management.

(The formula is not mine and is not taken on trust: the script computes it and checks it against the value published in the paper itself before printing anything else. If that check fails, everything downstream is void.)

---

So what counts as a “variant”?

The formula needs N. So we need to know what goes into it — and this is where the reasoning usually breaks.

N is not the size of your latest sweep. It is every configuration you have ever evaluated on this data.

Including:

• the sweep you ran last week and abandoned;
• the entry rule you replaced because it looked wrong;
• every time you nudged a threshold and re-ran;
• parameters you never formally optimised but adjusted after looking at a chart;
• what a colleague ran on the same history;
• the strategy you found on a forum, tested, and discarded.

All of it consumed one and the same finite sample. Data does not regenerate between runs. A grid of 60 combinations run after three months of exploration is not N = 60.

Check yourself on your current strategy right now: how many times have you run it? If the number surprises you, that is the finding.

---

“But my variants were nearly identical”

This is the strongest objection to the formula, and it is a fair one.

The formula treats variants as independent — it assumes their results are unrelated. But ten averaging windows between 18 and 22 periods produce nearly identical results. That is effectively one trial rather than ten, and the formula demands a correction you did not earn.

How much? I measured it. The table shows the actual best result among a hundred variants at varying degrees of similarity between them:

With very similar variants the formula overstates the correction threefold. That is a lot, and pretending the problem does not exist would be dishonest.

But notice two things.

Even nearly identical variants inflate the result. Bottom row: 0.79 against a true value of zero. That is enough to make an empty strategy look acceptable.

And the discount has to be earned, not asserted. Similarity between variants is measurable — you compute the correlation between their return series. It is not difficult. Saying “well, mine were all pretty similar” and lowering the correction on that basis is the same as not applying it at all.

---

Fine, so how do I test properly?

So far we have been discussing how to be wrong. Now, what to do about it.

There are two tests, and the main thing about them is that they answer different questions. They get confused and substituted for one another constantly, and that is an expensive mistake.

The permutation test

It asks: could a result like this have come from a rule with no connection to the data at all?

How it works: your sequence of trades is shuffled at random, many times over. Each shuffle produces a result. Together they form a distribution of what pure chance produces. Your result is then placed inside that distribution, and you can see whether it landed in the ordinary part of it or out in a distant tail.

What it does not do: it knows nothing about how many variants you tried. One run or a thousand, the test gives the same answer.

The Deflated Sharpe Ratio

It asks: given that I tried N variants and am showing you the best, what is the probability that its quality really is above zero?

How it works: it takes the threshold from the table above — the one expected from pure noise at your N — and compares your result not against zero but against that threshold. At the same time it corrects for the fact that the distribution of per-trade returns is not bell-shaped.

That second correction is not cosmetic on prediction markets. As the first article showed, per-trade returns on a binary contract are skewed anywhere from +2.3 to −3.1 depending on entry price alone. The ordinary Sharpe ratio ignores that skew, and at a 90¢ entry it ignores precisely the shape of risk you are holding.

Why both

A strategy can pass the permutation test and fail the DSR — that is the normal outcome for an overfitted sweep: the signal looks distinguishable from random, but not by enough to justify how hard you searched.

Passing both is the minimum. And it still does not prove the strategy works. It is the absence of two specific disqualifications.

---

If you want one number instead of tests

There is a simpler route: skip the DSR and just raise the significance bar.

The logic is the coins again. When there are many tests, the chance that at least one passes by accident goes up. To hold the overall risk of error at the same five percent, each individual test has to be made stricter.

About t — this is the quantity from the first table in the article. It shows how many times over your result exceeds its own margin of error. One means “a result the size of the error,” which is nothing. Two is the customary threshold beyond which a result is treated as significant. The formula is simple: t = Sharpe per trade × the square root of the number of trades.

The trade counts are computed at a Sharpe of 0.02 per trade — the figure that, in the first article, a genuine two-point edge produces at a 25¢ or 75¢ entry. At 50¢ the same edge yields 0.005, and the whole table multiplies by sixteen.

Why two columns of trade counts. The left one answers “when does the margin of error shrink to the size of the effect,” which is roughly a fifty-fifty chance of noticing the effect. Not “I will see it” but “I will see it half the time.” The right one is for the customary eighty percent. Use the right one.

Notice how differently the two columns grow. The |t| bar rises slowly: a hundredfold increase in variants costs about 1.8 sigma. But the trade count triples, because it is proportional to the square of t. Sweeping is cheap; data is expensive.

Harvey, Liu and Zhu reached the same conclusion for the published literature on return factors: accounting for the thousands of factors that were tested and never published, the customary |t| > 2.0 bar should be raised to roughly 3.0. Your parameter sweep is a private version of exactly the same problem.

---

What would refute this

The whole argument rests on one statement: the expected best result among N empty strategies grows with N. Run the script. If it did not grow, the correction would be unnecessary and this article would be wrong. Confirming it — by calculation and by simulation — takes under a minute.

The demonstration at the top is refuted the same way: replace the random walk with a series containing genuine predictability and see whether the right-hand column comes alive. If it does not, the simulation is broken and I want to know.

---

Limitations — the complete list

One family of strategies. “Momentum against a threshold” is not the only way to build a strategy, and a different sweep would give different numbers. The mechanism does not change, but the specific figures in the first table belong to this sweep.

The series is idealised. Real prices have fat tails and stretches of elevated volatility. Both effects tend to make overfitting worse rather than better, but that is not what I tested on.

The formula requires independent variants. Discussed above; with similar variants the correction is overstated, and that has to be measured.

Trades are treated as independent. The same assumption runs through the whole series. If neighbouring trades’ results are related — and on short-horizon markets they often are — the effective number of observations is smaller than the actual count, and every required sample size here is understated.

Passing these corrections proves nothing. Neither the DSR nor the permutation test is evidence of skill. They eliminate one specific way of being wrong.

Cross-validation schemes are not covered. There are stricter ways to split data than “in half” or “by rolling window.” They deserve their own treatment.

---

What to do with this tomorrow

1. Count N honestly and write it down before you look at the result. Not the grid size — every configuration that has ever touched this data.
2. Publish N alongside the result. A Sharpe without a variant count is an unlabelled number. Treat someone else’s — a colleague’s, a vendor’s, a forum post’s — the same way, and ask for N first.
3. Open the check period exactly once. Looked, disliked it, adjusted parameters, looked again — it has stopped being a check period, and N has grown.
4. Compare your trade count against the table. If you have two hundred and fifty trades per variant, you are in the top row of this article’s first table, and a positive result there means nothing.
5. Decide how many variants you will try before you start trying them. Each additional one permanently raises the chance of a false finding. This is the only part of the process that is free to fix in advance and impossible to fix afterwards.

---

<!-- ══ Cut this section for syndicated versions (§4 of the series plan) ══ -->

How this works in GoGoBots

The previous edition of this article carried an open question here: the platform computes a DSR, but it was unclear what N was being fed into it. That question is now closed — the counter is built. Here is what it does and, more importantly, what it does not do.

Configurations are counted, not runs. Run the same setup twice and N does not move. A repeat run is deterministic and carries no new information, so penalising you for reloading the page would be pointless.

Shifting the period does not reset the counter. Tested on January–March, then on February–April? That is the same data consumed twice. If the period were part of the key, the correction could be dodged by moving one date — and it would be.

Changing strategy does not reset it either. Three different ideas on one dataset consumed the sample three times over. The counter is attached to the data, not to your intent.

Both routes are counted — the agent and manual runs. There was a temptation to count only the AI agent’s sweeps, where N is reliable and large. But that would have produced a backwards incentive: identical behaviour judged differently. Forty configurations through the agent would get an honest DSR, forty manual clicks an optimistic PSR, and dodging the correction would quickly become a known trick.

Worse is the mixed case, which is also the most common: the agent sweeps two hundred variants, then a human takes the winner and hand-tunes it ten more times. If manual runs went uncounted, the accumulated correction would zero out precisely on the final candidate — the one heading for live trading.

The threshold uses the actual spread of your own results. The formula needs more than N; it needs how widely the Sharpe ratios of different variants diverge. Many implementations just plug in a one. We store the Sharpe of every trial and compute the spread from the data — at a spread of 0.5 the threshold is 1.27, at 1.0 it is 2.53, and the relationship is linear. An error here is the same order of magnitude as an error in N itself.

There is no reset, and N can only be raised. No event resets the counter automatically — any reset is a way around the correction. But you can raise N by hand, and that is not decoration: part of your search is invisible to the platform. Ideas discarded before implementation. Runs in someone else’s tool. Adjustments made after glancing at a chart.

Which is why the report reads “N ≥ 47” rather than “N = 47”. The wording is literally true, and it explains by itself why there is a manual input field next to it.

The check period is remembered. If you marked part of your data as a check period and then return to it a second time, the platform warns you that the window has stopped being out-of-sample, and increments N. This catches the most expensive process error there is — going back to the parameters after a disappointing check — by the only means that ever catches it: a reminder at the moment of the act.

What the counter does not do. It does not measure how similar your variants are to each other. As shown above, a hundred nearly identical variants behave like two independent ones, and in such cases the correction is overstated. This is computable — through the correlation of return series — but we do not compute it yet, and the interface says so plainly.

And it still does not know your full N. Nobody does.

How to check this without taking our word for it. Two minutes: run one configuration twice — N should not move. Shift the period by a week and run again — N should go up by one, not reset to zero. If it behaves differently, we have a bug.

<!-- ══ END ══ -->

---

FAQ

I only tried twelve variants. Is that fine? Twelve variants at zero quality give an expected best Sharpe of about 1.6. That is enough to make a useless strategy look respectable. Twelve is not a small number here. One is a small number.

My variants were similar, they are not independent. Probably true, and the correction is indeed overstated — see the table above. But measure the similarity rather than asserting it: it is the correlation between the variants’ return series. An asserted discount is not a discount.

So optimisation is pointless? No. It means the result of optimisation is a hypothesis whose credibility must be reduced in proportion to how hard you searched. Searching harder is not free, and the price is computable.

What if I really do have a strong result — say a Sharpe of 3? State it alongside N. At a thousand variants the expected best result of pure noise is 3.26 — meaning your Sharpe of 3 is below what noise alone would have produced. The same number after a single pre-declared test is a serious finding. One number, two very different pieces of evidence.

I do not have much data. What should I do? Sweep less, not more. The first table shows it: on a small sample, the more variants you try, the higher the chance you pick noise. This is a rare case where less work produces a more reliable result.

Why not just use more history? You can and should — it is the single most effective remedy. But history has a limit: markets on Polymarket exist for a finite time, and data that is too old describes a different market regime. So the second line of defence is counting N.

---

Not financial advice. Past results do not predict future results. Trading prediction markets carries risk up to and including total loss of capital. Verify that trading on these venues is legal in your jurisdiction.

Sources. Bailey & López de Prado, The Deflated Sharpe Ratio, Journal of Portfolio Management 40(5), 2014. Bailey, Borwein, López de Prado & Zhu, Pseudo-Mathematics and Financial Charlatanism, Notices of the AMS 61(5), 2014. Bailey & López de Prado, The Sharpe Ratio Efficient Frontier, Journal of Risk 15(2), 2012. Harvey, Liu & Zhu, … and the Cross-Section of Expected Returns, Review of Financial Studies 29(1), 2016. Lo, The Statistics of Sharpe Ratios, Financial Analysts Journal 58(4), 2002. Šidák (1967) — the correction for multiple comparisons.