How I Discovered That Most of My Backtesting Decisions Were Coin Flips

This is a story about how I found out that most of the decisions in one of my research logs were coin flips, and about what I changed so that it wouldn’t happen again.

In September 2026, I ran the same research model eleven times. The code, the data and the settings were identical each time, and the only thing I changed was the random seed. The best of those eleven runs scored about twice as well as the worst.

That would just be a curiosity if I hadn’t spent months making decisions by comparing a single run against a single other run. When we went back through the research log for that model, 14 of the 21 decisions it recorded, mostly about whether to keep a feature or throw it out, rested on a difference smaller than what the seed alone could produce.

If you’re new to this, here’s...

Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE