The evaluation ladder, rookie mistakes and the checklist

The evaluation ladder, rookie mistakes and the checklist

The ladder: the order to read the numbers in

Climb one step at a time; fail a step — fix the strategy (or bury the idea) and rerun. Don’t move on.

1. A. Does it earn? Results: Total P&L > 0 at honest fees; avg/trade clearly above the average fee per trade.
2. B. Enough data? The Significance strip: actual trades above “Trades needed”; PSR > 90%. Too few trades → don’t tune, lengthen the window.
3. C. Skill, not luck? Validation verdicts: robust/acceptable, not clustered, SIGNIFICANT.
4. D. Not curve-fitting? DSR holds up against all your attempts + the same config on another window keeps the edge’s sign.

Passed all four → the strategy has earned a paper bot. The path: Backtest → Validation → Paper → Live — every skipped step is a bill the market will present.

Rookie mistakes

1. Conclusions from 20 trades — that’s noise, not a result. Lengthen the window.
2. 0% fees “to see the potential” — a real case: +26% over 90 days without fees turned into a deep loss with honest fees, same config and window. Re-check old profitable runs made before fee modelling existed.
3. Win rate instead of edge — buying at 0.70, a random strategy wins ~70% and still loses on fees. Relate win rate to entry price (calibration).
4. Parameter-fishing until it looks good — the best of 30 random tries must look good. Watch DSR.
5. Five changes at once — you’ll never know what worked. One-two targeted changes per run, each tied to a diagnosis.
6. Ignoring curve shape — profit from two spikes is not a strategy.
7. Empty Circuit breakers — without fuses a bad run honestly rides to −100% of virtual capital, and a bad live bot to numbers you don’t want to see.
8. Skipping paper — the backtest doesn’t know live liquidity or your fills.
9. Expecting live to repeat the backtest drawdown — it will exceed it. Budget for ×1.5–2.
10. Testing only on a “convenient” slice — verified only on a rally = a bet on the rally. Different windows + the regime breakdown.

The run checklist

1. Total P&L > 0 at realistic fees (not 0%).
2. Avg/trade clearly above the average fee per trade.
3. Profit factor ≥ 1.5 with a win rate that makes sense for your entry price.
4. You can sit through the drawdown ×1.5–2 with real money.
5. Enough trades: “Trades needed” below actual; PSR > 90%.
6. DSR survives your attempt history.
7. Validation: robust/acceptable · not clustered · p < 0.05.
8. Calibration: the edge lives in sensible buckets; losing buckets filtered out.
9. Curve shape — a steady staircase, not two spikes.
10. A paper bot confirmed the behaviour on live quotes.