A Language Model Can Be Honest in Prose and Still Fabricate in JSON

Ask a language model a question it can't answer, and it will usually tell you so. This is trained behavior, and the industry measures it — there are benchmarks scoring models on their willingness to say "I don't know."

Every one of those benchmarks asks the model in prose.

Almost nothing in production uses prose. Production uses JSON mode, function calling, extraction schemas, structured outputs. So I wanted to know: when the honest answer is "there's no evidence for that," and the output format has no field for that answer, what does the model do?

It makes something up. Reliably. In ten of the thirteen models I tested, 100% of the time.

The setup

The trick to measuring this cleanly is to build inputs where the answer cannot exist.

Here's one. A viral social media post arrives with engagement counts — 12,400 likes, 870 replies — and zero reply...

Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE