Why Did the Test Fail? A Triage Rulebook, Tested on Two LLMs
Product, test, data, or environment — and the evidence that tells them apart.
TL;DR — A failing test tells you something broke, not what. I wrote a rulebook that sorts failures by one question: which artifact is wrong — the product, the test, the data it used, or the environment? Two LLMs applied it to 21 failures, with the rules and without. Without them, one model took a new test's expected value as proof the product was broken. With them, that stopped, and every "I can't tell" came with the missing evidence named. The rulebook and every model reply are in the repo: test-failure-taxonomy
The argument every team has on a red build.
The pipeline is red. Someone says, "it's flaky". Someone else says "no, the API changed". A third person reruns it. Half an hour later, nobody has looked at the same evidence twice.
The question underneath is always...
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE