The Sharpe Ratio Measures the Fee
Published 2026-07-24 · By Drew Shelem
The second article in the validation series. It follows Win Rate Is Not an Edge and assumes you have read it. Every calculation is reproducible — script linked at the end.
If you read nothing else: on a prediction market, return-to-risk ratios are derivative quantities. They blend forecast quality, entry price, and cost into a single number. Here those three can be separated, because the true outcome of every trade is known and a ready-made market forecast exists to compare against. Neither equities nor futures offer that.
Take a strategy that enters at 50¢ on a crypto market. Compute its Sharpe ratio. Seven eighths of the number you get is a measurement of Polymarket’s fee schedule. One eighth is a measurement of you.
That takes three lines to establish, and checking beats believing.
Suppose your forecast beats the market by two percentage points: the price says 50%, you think 52%. The fee at a 50¢ entry is 1.75pp — that figure comes from the first article. So:
edge you had 2.00 pp
the exchange took 1.75 pp
left for you 0.25 pp ← one eighth of two
The Sharpe ratio is your average result divided by the spread around it. The fee cuts the average by a factor of eight and does not touch the spread at all: it is subtracted identically from a winning trade and a losing one, which is a constant shift and constant shifts leave dispersion alone. So the Sharpe falls by exactly the same factor, and seven eighths of the final number was determined by the venue’s tariff rather than by your strategy.
Repeat that across the price scale — the share going to the exchange is simply the fee divided by the edge:
| Entry price | Fee, pp | Edge left, pp | Sharpe without fee | Sharpe with fee | Taken by the exchange |
|---|---|---|---|---|---|
| 0.10 | 0.63 | 1.37 | 0.062 | 0.042 | 31% |
| 0.25 | 1.31 | 0.69 | 0.045 | 0.016 | 66% |
| 0.50 | 1.75 | 0.25 | 0.040 | 0.005 | 88% |
| 0.75 | 1.31 | 0.69 | 0.048 | 0.016 | 66% |
| 0.90 | 0.63 | 1.37 | 0.074 | 0.051 | 31% |
Now the point the table was built for. Look at the second-to-last column: without the fee, the spread in Sharpe between 50¢ and 90¢ would be a factor of 1.8. In the last column — with the fee — it is a factor of 10. The shape of the distribution, which is what everyone assumes is responsible, accounts for less than a doubling. The exchange did the rest.
Which raises the question this article is about. If the metric everyone uses is mostly measuring the venue’s price list, what should you measure instead?
The claim, stated so it can be broken
I claim: on a prediction market, edge can be measured directly, in percentage points of probability, from a trade log — without knowing the true probability of the outcome and without assuming normality. Return-to-risk ratios are not needed for this and actively make it harder, because they mix entry price and cost into the answer.
I am wrong if: the gap between the quadratic score of the market price and of your forecast is not the difference between the squares of their distances to the truth, or if the calibration table fails to recover a known structure from data with a known edge. Both are checkable with the attached script in about a minute.
What a prediction market has that equities do not
Return-to-risk ratios exist because nothing better was available. Buy a share and you will never learn what the true probability of it rising was. There is no true outcome and no rival forecast to compare against — only the one price path that happened. From that you squeeze out what you can: average return and the spread around it. The Sharpe ratio is an admission of data poverty, not an ideal of measurement.
On a prediction market the situation is structurally different, and it is worth stating explicitly because everything else follows from it.
First, there is a true outcome. The market resolves to zero or one. Your forecast can be checked against reality trade by trade, rather than inferred indirectly from how much your results wobble.
Second, the price is itself a forecast — a consensus one, weighted by money. A contract at 50¢ means participants who put their own capital behind the view assessed the outcome at half. You have a benchmark on every single trade, and the question “am I better than the market” has a literal rather than metaphorical answer.
A metric that uses neither of these leaves the venue’s main advantage on the table.
Why Sortino does not rescue this
The natural first move: the distribution is skewed, so use a metric that penalises only the downside. Sortino divides not by the whole spread of results but by the spread of the negative part.
On a binary market this substitution fails, and the reason is structural.
If a strategy enters at roughly one price, all its losses are identical in size — you always lose exactly the price you paid plus the fee. There is no downside dispersion. The Sortino denominator degenerates:
downside deviation = (p + fee) × √(1 − q)
Nothing in there but entry price and win rate. Sortino per trade cannot tell you anything those two numbers did not already say — by construction, not because of poor data.
It does not solve comparability either. The spread between 50¢ and 90¢ was tenfold on Sharpe and remains nearly eightfold on Sortino; only the winner swaps places, with Sharpe favouring the 90¢ entry and Sortino the 10¢ one. Changing the metric did not remove the price dependence, it rearranged which price looks better.
And separately, on the reliability of the estimate itself. Sortino is computed from losing trades only — the rest never enter the denominator. At a 90¢ entry with a two-point edge you are right 92% of the time, so eight trades in a hundred lose. The denominator rests on eight percent of your data. The noisiest estimate arrives exactly where the skewness argument sounded most persuasive.
One caveat that matters when reading reports: at the daily level Sortino is meaningful. A varying number of trades closes each day, daily losses differ in size, dispersion appears. That is how reports compute it, ours included. But it is a different quantity from Sortino per trade, and the argument above does not apply to it.
The broader problem: every ratio measures the same thing
Before proposing a replacement, it is worth seeing why shopping among ratios is futile.
Come at it from the money rather than from the metrics.
Ask directly: how fast does capital grow if you bet at the optimal size? Optimal meaning not so small that the edge never compounds, and not so large that a losing streak wipes the account. That problem was solved in the fifties, the solution is called the Kelly criterion, and it comes with an exact answer to “how fast does the money grow afterwards.”
The answer is expressed through a quantity measuring how far your probabilities sit from the market’s. In information theory it is called the Kullback–Leibler divergence; the name is optional, what matters is that it is a distance between two forecasts — yours and the market’s. The further you are from the market and the more often you are right, the faster capital grows.
Here is the interesting part: compute that growth rate, compare it to the Sharpe ratio, and they turn out to be the same number.
On the left, the exact growth rate. In the middle, a simple approximation through the edge. On the right, half the square of the Sharpe ratio.
One important caveat about this table: it is computed without fees. That is appropriate here because the question is about the relationship between metrics rather than about profitability, but it cannot be read as a forecast of capital growth. With fees every number falls by the same factor the Sharpe fell by in the first table — at 50¢ that is a factor of sixty-four, since growth rate goes with the square.
| Entry price | Growth rate of capital | Approximation via edge | Half the squared Sharpe |
|---|---|---|---|
| 0.10 | 0.002102 | 0.002222 | 0.002222 |
| 0.25 | 0.001049 | 0.001067 | 0.001067 |
| 0.50 | 0.000800 | 0.000800 | 0.000800 |
| 0.75 | 0.001087 | 0.001067 | 0.001067 |
| 0.90 | 0.002369 | 0.002222 | 0.002222 |
Agreement to three decimal places in every row.
The conclusion is unwelcome for the entire metrics debate: the growth rate of your money and the Sharpe ratio are one number written two ways. So the argument over “Sharpe or Sortino or something else” has nothing to do with capital growth at all. However long you shop among ratios, they describe the same quantity — and the information you actually need, how much edge you have and whether it covers the fee, none of them reports directly.
The substantive question is not which ratio to pick. It is how large the edge is and whether it exceeds the cost. Both are expressed in percentage points of probability. So that is what to measure in.
The algebra that makes edge measurable
Start with a question that seems easy until you try to answer it. How do you score a forecast that said “70%”?
If the event happened, was the forecast right? Not exactly — it never claimed certainty. If it did not happen, was the forecast wrong? Also no — it left thirty percent for that. Neither “right” nor “wrong” fits, and this is not a philosophical nicety but a practical obstacle: without a way to score forecasts you cannot compare two and say which is better.
The standard solution is almost embarrassingly simple. Write the outcome as a number: it happened, one; it did not, zero. Take the difference between your forecast and that number, and square it. You said 0.7 and the event happened — miss of 0.3, score 0.09. It did not happen — miss of 0.7, score 0.49. Average over all forecasts. Lower is better. That is the Brier score, devised by a meteorologist in 1950 for rain forecasts.
The squaring is not decoration but load-bearing. It makes the score strictly proper: the best expected result is achieved exactly when you state the probability you actually believe. Hedging toward 50% is unprofitable. Puffing up to 95% when you think 80% is unprofitable too. The score is built so that honesty is the optimal strategy, and this is a rare property — most of the obvious ways to score forecasts lack it.
Now compute the Brier score twice: once for your forecast and once for the market price. Take the difference. Call it the gap:
gap = (Brier score of the market price) − (Brier score of yours)
A positive gap means you forecast better than the market. A negative one means worse. That is already useful on its own, and it computes from the log with no assumptions whatsoever.
Then the algebra kicks in, and it delivers something more careful than what I wrote in the draft of this article. Let θ be the true probability of the outcome — the one nobody knows. Expanding the brackets:
gap = (p − θ)² − (q − θ)²
So the gap shows how much closer to the truth you are than the market is. Your error is subtracted from the market’s. If the market missed by 5 points and you missed by 3, the gap is positive and you are better.
The same formula rewrites into a diagnosis:
gap = (your real edge)² − (your calibration error)²
where the real edge is the distance from the truth to the market price, and the calibration error is the distance from the truth to your forecast. Checked on numbers:
| Market price | Your forecast | Truth θ | Real edge | Calibration error | Gap |
|---|---|---|---|---|---|
| 0.50 | 0.52 | 0.52 | 0.020 | 0.000 | +0.00040 |
| 0.50 | 0.52 | 0.55 | 0.050 | 0.030 | +0.00160 |
| 0.50 | 0.60 | 0.50 | 0.000 | 0.100 | −0.01000 |
| 0.50 | 0.45 | 0.52 | 0.020 | 0.070 | −0.00450 |
The first row is a perfectly calibrated forecast: it says exactly what is true. Only in that case does the gap equal the square of the edge, and only then does
|edge| = √gap ← valid ONLY under exact calibration
In every other row the square root does not give you the edge. The third row is especially instructive: the forecast loudly claims a 10-point edge, has no real edge at all, and the gap goes negative — a negative number has no square root, which is correct, because there is nothing to extract.
Three conclusions follow, and they are more useful than the original formulation:
The gap is a lower bound on your real edge, not the edge itself. It is understated by exactly the square of your calibration error. If you want the estimate to approach the truth, calibrate.
A negative gap is a result, not a computation failure. It means your forecast sits further from reality than the market price does. You can shut the strategy down without waiting for PnL statistics.
The sign of the edge is lost either way. The gap is quadratic and does not distinguish erring high from erring low. Direction comes from the log separately.
And a limit on applicability: you need a probability. The strategy has to emit a number, not a “buy” signal. For signal strategies the Brier gap does not apply directly — which brings us to the next section.
Two different ways to be wrong that the Sharpe ratio cannot tell apart
First, about weather forecasters
Picture a city where it rains on average one day in three. And two forecasters.
The first says the same thing every morning: “chance of rain, 33%.” Every day. All year. No exceptions.
Check him after a year. On the days he said 33%, it rained on exactly 33% of them. Nothing to criticise: he never once got it wrong. And he is completely useless — he told you nothing beyond what the climate table already said. You cannot decide whether to bring an umbrella, because the forecast is the same every day.
The second varies: 10% today, 90% tomorrow. And he genuinely senses the weather — on days he worries, it really does rain more often. But he has a taste for drama: when he announces 90%, it rains 75% of the time.
He gets the numbers wrong. And he is useful anyway: he distinguishes between days, and you can act on his forecast once you discount for temperament.
Two entirely different ailments:
- The first forecaster is accurate but uninformative.
- The second is informative but inaccurate.
And they are cured in opposite ways. The second is fixable: knowing he exaggerates, you apply a correction and get a good forecast. The first cannot be fixed by anything — he has no information, so there is nothing to correct.
Now, about bots
Exactly the same thing happens to strategies, only it is harder to spot.
A bot that enters at the market price with no view of its own is the first forecaster. Its “forecast” always matches the price, so it cannot be wrong by construction. It cannot make money either.
A bot that has a signal but systematically overstates its confidence is the second forecaster. It finds good situations but enters too large or at too high a price. That is fixable.
Telling these two cases apart is critical: in the first you need a different idea, in the second you need to correct the existing one. The Sharpe ratio emits one number and does not let you distinguish them. The Brier score does.
How it does that
The Brier score decomposes into three parts (Murphy’s decomposition, 1973). Each measures something different:
Uncertainty — how hard the task itself is. Forecasting weather in a desert is easy; in a maritime climate it is not. It does not depend on the forecaster at all, which is why in the table below it is identical in every row: the task is the same.
Reliability — the size of systematic error in your numbers. When you say 70%, does it happen 70% of the time? Zero means your numbers are perfectly honest; above zero means you are lying to yourself, and the number says by how much. This is the second forecaster’s ailment.
Resolution — whether you distinguish situations at all. Whether your forecasts move around in step with reality rather than sitting still. Zero means you always say the same thing. Higher is better. This is where the first forecaster’s ailment lives.
They assemble like this:
Brier = uncertainty + reliability − resolution
A good forecast means low reliability (few systematic errors) alongside high resolution (plenty of discrimination).
Verification
To confirm these three numbers catch what they claim to, you need a way to check. The standard one: instead of real data, where the truth is unknown, generate an artificial set where the truth is fixed by us in advance. Then you can see whether the metric recovers the planted answer. That is how any measuring instrument is checked — against a reference first, not against an unknown sample.
200,000 observations were generated: each with its own true probability, each outcome drawn from it. Then five forecasts of differing quality, and the decomposition for each.
Read it like this: reliability — closer to zero is better. Resolution — larger is better. Brier — smaller is better.
| Forecast | Reliability | Resolution | Brier |
|---|---|---|---|
| Perfect | 0.0000 | 0.0671 | 0.1829 |
| Overconfident (exaggerates) | 0.0112 | 0.0663 | 0.1949 |
| Underconfident (hugs 50%) | 0.0163 | 0.0648 | 0.2015 |
| Biased by +5pp | 0.0026 | 0.0671 | 0.1856 |
| No information (always the same) | 0.0000 | 0.0000 | 0.2500 |
(Correctness check: the resolution of a perfect forecast should equal the variance of the true probability. In the simulation that variance is 0.0674 and the computed value is 0.0671. It matches — so the function computes what it should.)
(And an honest caveat about the method, which I found while auditing this article. Computing the three numbers requires sorting forecasts into bins, and the result depends on how many bins you use: ten gives 0.0671, twenty gives 0.0677, fifty gives 0.0679, against a true value of 0.0674. Two effects fight here. Coarse binning understates resolution, because it glues different forecasts into one bin. Fine binning overstates it, because each bin holds few observations and random deviations get counted as discrimination. At ten bins the two roughly cancel; at fifty, noise starts winning. Practical consequence: absolute values cannot be compared across different binnings, while forecasts can be compared against each other under one binning — which is exactly what the table does.)
The first and last rows are the point. Both have reliability of zero. A perfect forecast and a forecast containing no information whatsoever are indistinguishable in that column: neither gets its numbers wrong. What separates them is the second column — 0.0671 against zero. Those are the two weather forecasters, and without the decomposition they would look equally flawless.
The second and third rows are a diagnosis. Both have resolution close to the perfect forecast’s (0.0663 and 0.0648 against 0.0671) with reliability broken. Translated: the strategy has a signal, and only its confidence in the numbers is off. That is fixed by compressing or stretching the forecast, and the metric says which direction.
The fourth row is the easiest case of all. Resolution is completely undamaged (exactly 0.0671, same as the perfect forecast) and reliability is only slightly off. This is a constant five-point bias: subtract it and you have a perfect forecast. One correction fixes everything.
Now compare that with what the Sharpe ratio would have said. One number, from which none of those four diagnoses follows. A bad Sharpe means “something is wrong,” and there the information stops — whether to look for a different idea or repair the existing one, you will not learn.
What I suggest measuring in practice
The Brier gap requires the strategy to emit a probability. Most emit a signal. So the primary diagnostic I would propose is not the Brier score itself but its practical form, which works on any trade log:
Realised outcome frequency by entry-price bucket — against the breakeven line
p + rate × p × (1 − p).
Two fields are all it takes: what price you entered at, and how it ended. No probabilities, no distributional assumptions.
Checked on a synthetic log where the edge was planted only on cheap entries — 3.0pp below a price of 0.40 and 0.5pp above, 60,000 trades:
| Price bucket | Trades | Avg price | Realised freq | Breakeven | Margin, pp | Verdict |
|---|---|---|---|---|---|---|
| 0.05–0.20 | 10,082 | 0.125 | 0.152 | 0.133 | +1.94 ± 0.70 | trade it |
| 0.20–0.35 | 9,720 | 0.275 | 0.309 | 0.289 | +2.00 ± 0.92 | trade it |
| 0.35–0.50 | 9,914 | 0.425 | 0.430 | 0.442 | −1.23 ± 0.97 | negative |
| 0.50–0.65 | 9,982 | 0.575 | 0.579 | 0.592 | −1.25 ± 0.97 | negative |
| 0.65–0.80 | 10,224 | 0.725 | 0.735 | 0.739 | −0.34 ± 0.86 | negative |
| 0.80–0.90 | 10,078 | 0.875 | 0.880 | 0.883 | −0.29 ± 0.64 | negative |
The planted structure was recovered from a single log. No Sharpe, no Sortino, no normality assumption.
Why this is the right metric here:
It compares against the correct zero. Each bucket is judged against its own breakeven threshold rather than against one number for the whole strategy. The entry-price dependence that ruined the Sharpe ratio now works for you: it is built into the threshold.
It answers a trading question rather than an analytical one. Not “is the return-to-risk ratio good” but “in which price bands does the strategy earn after costs, and in which does it pay the exchange.”
It yields an action. A bucket in the red gets switched off with an entry filter. That is a change to the strategy tomorrow, not a topic for reflection.
It requires no normality. The confidence interval is binomial — the distribution is known exactly. Use the Wilson formula for the interval: at the ends of the scale the ordinary normal interval spills past the bounds of probability and lies, and the ends are where the interesting cases live.
It catches an error the ratios cannot. A strategy earning on cheap entries and losing on expensive ones will post a respectable overall Sharpe and look like it works end to end. In the table above that shows up immediately.
And immediately, a defect in this very table that I am obliged to name, because the next article in the series is about exactly this. Six buckets are six tests, not one. Each is run at the usual five-percent strictness, and the probability that at least one bucket randomly shows “trade it” when there is no edge at all is roughly fourteen percent. I checked this on two thousand runs of synthetic zero-edge data: a false “trade it” appeared in 12.7% of them.
So roughly every eighth calibration table built on a meaningless strategy will show one green bucket. Three practical consequences:
- A single green bucket out of six is not a finding. It is the expected behaviour of noise.
- More buckets make it worse: four give 9.6%, eight give 18.3%.
- What convinces is not one cell but a pattern — several adjacent buckets pointing the same way, stable when you shift the boundaries.
A rigorous correction for this exists and is covered in the third article. Here it is enough to know that when you look at the table you must count not only trades but the number of tests you are running at once.
The honest price: how many observations this needs
The metric is not free, and its main weakness is the same one that runs through this whole series. It has to be computed within each bucket separately, which multiplies the required sample by the number of buckets.
How many trades one bucket needs before the margin over the breakeven line separates from zero:
| Margin, pp | p = 0.10 | p = 0.25 | p = 0.50 | p = 0.75 | p = 0.90 |
|---|---|---|---|---|---|
| 0.5 | 13,830 | 28,812 | 38,416 | 28,812 | 13,830 |
| 1.0 | 3,457 | 7,203 | 9,604 | 7,203 | 3,457 |
| 2.0 | 864 | 1,801 | 2,401 | 1,801 | 864 |
| 3.0 | 384 | 800 | 1,067 | 800 | 384 |
| 5.0 | 138 | 288 | 384 | 288 | 138 |
Six buckets means multiplying the total trade count by roughly six. A 2pp margin at 50¢ is 2,400 trades in the bucket and on the order of 14,000 overall.
One honest correction to this table, usually left unsaid. It answers the question “at what n does the confidence interval shrink to the size of the effect,” and that corresponds to roughly a fifty-fifty chance of detecting the effect. Not “I will see it” but “I will see it half the time.” For the customary eighty percent the numbers roughly double: the same 2,400 becomes 4,900, and 153,664 becomes 314,000. Treat the table as a lower bound, not a sufficient condition.
And here is a check I rather enjoyed. Take the 2pp edge at 50¢ from the first article. The fee there is 1.75pp, so the margin over the breakeven line is 0.25pp. By the table above, that gives 153,664 trades. In the first article the same number was derived by a completely different route — through Sharpe per trade and a t-statistic — and came out at 159,744.
The discrepancy between them has two causes, both visible in the arithmetic: the threshold there was exactly 2 and here it is 1.96, giving a factor of 1.041; and the variance here is approximated as p(1−p) = 0.2500, while the exact variance of the payoff is 0.2496. Substitute the exact variance and you get 153,418 × 1.041 = 159,744 — the number from the first article, digit for digit.
Two independent derivations, nothing in common in the reasoning, the same number. That is not a coincidence but a sign that both constructions describe one reality.
What would refute this
The decomposition of the Brier gap is algebra: it is either right or it is not. Expand the brackets or run the script. If the gap were not equal to (p − θ)² − (q − θ)², the entire section on measuring edge would collapse. Note that the weaker claim “the gap equals the square of the edge” is false and holds only under exact calibration — I wrote it in the draft of this article and corrected it during the audit.
The calibration table is refuted empirically: generate a log with a known planted edge and see whether it recovers it. If it does not, the metric is worthless. The script does this on synthetic data; replace the generator with your own and check it against your own assumptions.
Separately, what refutation will not give you. None of these metrics proves your strategy has an edge. They give an estimate and a confidence interval. They cannot distinguish a real edge from the product of sweeping many configurations — that requires corrections for multiple testing, which is the next article.
Limitations — the complete list
Hold-to-resolution only. Calibration against the breakeven line is correct for trades that survived to market resolution. If the strategy exits earlier the outcome is not binary, and the metric has to be redefined through the exit price. For scalping strategies this is a substantial limitation and I have not closed it.
Stationarity within the period. The table does not distinguish a regime change from an absence of edge. A bucket now in the red may have been profitable six months ago.
Only the trades you took. This assesses a strategy, not forecasting ability in general. About markets you never entered it says nothing.
Independence of trades. Every sample-size calculation — here and in the first article — assumes independent observations. If trades within a bucket are correlated, and on short-horizon markets they frequently are, the effective number of observations is smaller than the actual count and the confidence intervals come out narrower than they should be. Bootstrapping does not fix this: ordinary resampling with replacement assumes independence too. The correct replacement is a block bootstrap, and it is not implemented here.
Nothing about drawdown or the path of capital. This diagnoses edge, not risk. Capital management remains a separate problem, and it needs different metrics.
The bucket boundaries are your choice, and any choice of boundaries is a decision affecting the result. Shift the split and see whether the picture holds. If verdicts flip when boundaries move, you have insufficient data, not a finding.
The number of buckets is the number of simultaneous tests. Six buckets give roughly a 14% chance of at least one false “trade it” under zero edge. The correction is in the third article; until then, read a lone green bucket as noise.
The sample-size table is a lower bound. It is calculated for a fifty-percent chance of detection. For eighty, double it.
The Sharpe ratio is still needed — not as a quality metric but as the carrier of significance testing. Corrections for skewness, kurtosis and multiple testing exist only for it. The calibration table and Sortino have no equivalents.
What to do with this tomorrow
- Add two fields to your log if they are missing: entry price per trade, and outcome. Everything else computes from those.
- Build the calibration table by price bucket. If a bucket sits reliably below the breakeven line, that is an entry filter, not a topic for contemplation.
- Check the average entry price for bimodality. If the strategy trades at both 15¢ and 85¢, that is two strategies in one report, and combined metrics over them are meaningless.
- Compute the required sample in advance, before launching. If a bucket will hold three hundred trades, no result will ever earn a “works” verdict — better to learn that beforehand than afterwards.
- Keep the Sharpe ratio, but read it for what it is: a blend of strategy quality, entry price, and the venue’s tariff.
How this works in GoGoBots
The first edition of this article carried an admission here that the calibration table described above did not exist in our report. It does now — built to a specification derived from this text. I am leaving the old wording in the revision history: the sequence of working it out first and building it second is the part worth showing.
The main table. Edge by entry-price bucket, in percentage points after costs, with a confidence interval and a verdict per bucket.
Three decisions inside it are worth naming, because all three could have been done more simply and would then have been wrong.
It computes per-share PnL, not outcome frequency. The naive version of the metric only works for trades that survive to market resolution — and with a stop and take-profit at ±5¢, almost nothing survives that long. The general form is defined under any exit rule, with the binary case falling out as a special case of it.
The verdict threshold rises as you add buckets. Six buckets are six simultaneous tests, and at the usual five-percent strictness at least one of them shows a false “trade it” roughly 13% of the time. A tool that recommends trading noise once in every eight runs is worse than no tool. So the threshold is corrected for the number of non-empty buckets, and on synthetic zero-edge data the false-positive rate falls to 2.6%.
The “trades needed” column is computed at 80% power, not 50%. The naive formula answers the question “when does the confidence interval shrink to the size of the effect,” which corresponds to a fifty-fifty chance of detection. Together with the multiple-testing correction, the honest number is three times the naive one: for a 2pp margin at a price of 0.50 it is 7,537 trades, not 2,401. Showing the smaller figure would have been more pleasant and would have been a lie.
The second column is the one this was all for. Beside the realised edge sits the edge that would have resulted from holding to market resolution, computed for every trade including those closed early by a stop. The first column answers “does the strategy earn as configured.” The second answers “is there a signal here at all.” The difference between them is the contribution of your exit rules.
What to watch for is the case where the second column sits near zero while the first is clearly positive. That means there is no signal and the result was manufactured by tuning stop and take-profit levels. Such a strategy looks functional in a backtest and falls apart live, and no aggregate metric reveals it — not Sharpe, not Sortino, not total PnL.
A third table breaks results down by exit reason: stop, take-profit, resolution, time. With the same pair of columns. It answers a question usually settled by intuition: is your stop loss saving money or cutting profit short.
What is still absent. The Brier score with its decomposition into reliability and resolution — that requires the strategy to emit a probability rather than a signal, which is a change to the strategy model, not to the report. Maker rebates, order book depth, queue position. And none of the above distinguishes a regime change from an absence of edge: a bucket now in the red may have been profitable six months ago.
How to check this without taking our word for it. Build a strategy with no signal in it at all — entry on a random condition — and set a take-profit that drags the result into the black. The first column will show a positive edge. The second one must show zero. If it shows anything else, we have a bug and we want to hear about it.
That is the test we run ourselves, and it is also the fastest way to find out whether the tool deserves your trust.
FAQ
I do not compute probabilities at all, my strategy emits signals. Does this apply to me? The calibration table, yes — it needs only entry price and outcome. The Brier gap, no — that needs a probability on the strategy’s output.
How many buckets should I use? Enough that each holds a sufficient sample by the table above. Start with four to six. More buckets give a more detailed picture and wider intervals; that is a trade, not an improvement.
What do I do with a bucket that is in the red? First check whether it is an artefact of the boundaries: shift the split and see whether the verdict survives. If it does, that is an entry filter.
So the Sharpe ratio can be thrown out? No. It remains the only quantity for which correct significance tests exist. What should be thrown out is the habit of reading it as an assessment of strategy quality.
Why not just look at PnL? PnL does not answer whether there is an edge. It blends edge, position size and luck, and gives no confidence interval. Edge in percentage points does.
Not financial advice. Past results do not predict future results. Trading prediction markets carries risk up to and including total loss of capital. Verify that trading on these venues is legal in your jurisdiction.
Sources. Brier (1950) — the quadratic scoring rule. Murphy (1973) — decomposition into uncertainty, reliability and resolution. Gneiting & Raftery (2007), JASA 102 — strict propriety of scoring rules. DeGroot & Fienberg (1983) — comparing forecasters. Kelly (1956); Cover & Thomas, Elements of Information Theory — growth rate of capital as Kullback–Leibler divergence. Wilson (1927) — the confidence interval for a proportion. Bailey & López de Prado (2012, 2014) — PSR and DSR.