Your Health Checks Are Verifying the Wrong Thing
Four bugs in one week. All of them passed monitoring. All of them were the same bug.
Last week I found four separate faults in a system I built. A financial figure that was wrong by 16 times. A metric frozen for a month while appearing to update. Nine decommissioned processes still holding live credentials. And 149 backup files serving secrets over HTTP to anyone who asked.
Every one of them passed monitoring the whole time. Not because monitoring was broken. Because every check was verifying that something existed rather than that it was true.
That distinction is the whole article. If you run scheduled jobs, agents, or anything that reports its own health, you have written this bug. I can almost promise it.
The first one, told properly
We report monthly recurring revenue on an internal dashboard. It showed a number. The number was wrong by a factor of...
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE
