Validation — luck or skill

업데이트: 2026-08-14

A pretty P&L is necessary but not sufficient. Validation answers the key question: “did I just get lucky with this slice of history?” For decisions, the verdicts are enough; here is what each test does.

Trade bootstrap — Monte Carlo over trades

Your trades are a bag of chips; we assemble 1000 random “alternative histories” from them. On screen — a fan of 50 curves (the bold one is your real run) and tiles:

Profitable — share of simulations ending positive (95%+ is great);
Fragility — share ending at/below zero: robust (<15%) · acceptable (<30%) · fragile;
Sharpe med / 90% CI — median and interval of simulated Sharpes;
Beat orig. — share beating your actual result (nearly all worse = your real ordering was unusually lucky).

Order-shuffle — reshuffling the order

Total P&L doesn’t depend on trade order — drawdown very much does. 2000 shuffles: Percentile — the share of shuffles with drawdown no worse than yours. Verdicts: normal · elevated (≥80) · clustered (≥95) — your losses clump in time; in bad periods the strategy loses serially and the real drawdown will run deeper. The same is computed for drawdown duration in days.

Block bootstrap

Appears when autocorrelation (below) is detected: plain bootstrap breaks up streaks and understates risk, the block version reassembles history in contiguous chunks. If its verdict is worse — trust the worse one.

Significance & autocorrelation

p-value — chance of such average daily returns by luck; SIGNIFICANT at p < 0.05 (NOT SIG = “not enough data”, not a verdict of failure);
Sharpe 95% CI — does the interval cover zero;
Min trades needed — sample size required at the current edge strength;
Ljung–Box — do results clump into streaks (CORRELATED = wins/losses come in waves → check block bootstrap; drawdowns will drag).

Benchmark baselines

Comparison against “dumb” baselines: how often UP won at all, what “buy UP and hold” would have made (and DOWN), and alpha — your excess over that. If you can’t beat buy-and-hold, the complex rules aren’t paying for themselves yet.

Technical footnote: Sharpe here is annualised ×√252 while the Results headline uses ×√365, so the numbers may differ slightly. Different rulers, not a bug: treat the Results Sharpe as canonical and read verdicts/intervals here.