2026-06-25 · Brent Akamine
The placebo test: the one gate that retires the most strategies
There is a question almost no published backtest can answer: is this better than random?
Not “did it make money” — a coin flip makes money half the time. Not “did it beat buy-and-hold” — that’s a different bet. The question is whether the signal — the specific rule you believe in — carries information that random labels on the same data would not.
That is what the placebo gate measures, and it is the single most decisive test in The Validation Gauntlet.
How the placebo works
Take the strategy. Keep the exact same data, the same costs, the same number of trades, the same position sizes. Now shuffle the thing you believe in — randomly permute the trade-direction labels, or randomly flip the sign of each signal — and re-run. Do it 200 or more times. You get a distribution of profit factors that the machinery produces when the signal is noise.
Then ask one thing: does the real strategy’s profit factor beat the 95th percentile of that random distribution?
If yes, the signal is doing something random labels can’t. If no — if a third of random permutations do as well as your “edge” — then whatever P/L you have is not attributable to the signal. You have a backtest-shaped object, not a strategy.
What it has caught
The placebo gate has been the executioner more often than any other gate in the program:
-
The carry trade. Hardened, pre-registered, run over 15 years of real data, cross-sectional G8 carry posted a profit factor of 0.844 — and 33% of random sign-permutations beat it. The real strategy sat below the placebo 95th percentile. Ranking currencies by interest rate was worse than random over 2011–2026. Full report →
-
ETH/BTC pairs trading. A cointegration mean-reversion spread trade that looked plausible in-sample was beaten by 58% of random-sign permutations. The cointegration relationship that justified the trade did not produce a tradeable edge. Full report →
And, just as importantly, what it has confirmed:
- The Turtle breakout beats its random-breakout placebo at the 99th percentile — only 0.8% of permutations match it. Real edge →
- The funding-rate harvest beats its placebo so decisively (95th percentile 0.003; 0% of 500 shuffles match) that we kept investigating why it still failed other gates — and found a cost problem, not a signal problem. Real edge, wrong rule →
Why almost nobody runs it
Because it is designed to disappoint you. The placebo doesn’t care how clever your rule is or how clean the equity curve looks; it asks whether the rule beats its own shuffled shadow. Most strategies don’t, and most strategy-sellers would rather not know.
That’s exactly why we run it on everything, and publish the result either way. A strategy that survives the placebo is rare — and that rarity is what makes it worth something. It’s the difference between an edge and a story about an edge.
See every strategy’s placebo result on the scoreboard.
By Brent Akamine (Founder, Vinovest). Part of The Validation Gauntlet. Backtests are not investment advice.