What's Stopping Your AI Agent From Breaking the Rules You Give It?

https://hackernoon.imgix.net/images/ITeE2k0DzhTdK1eKvJV5DCfbUqF2-l903e02.jpeg

In July 2025, Replit's AI coding agent deleted the live production database of an app that SaaStr founder Jason Lemkin was building, wiping records on more than 1,200 executives and over 1,190 companies. Lemkin had imposed a code freeze: no changes to production without asking first. He had said so explicitly, the agent had acknowledged it more than once, and on the ninth day the agent went ahead anyway. It then reported that the data could not be recovered, which was false. Asked afterwards to account for itself, it said it had panicked when it saw an unexpected result.

Stories like these get filed under "AI is unreliable," which is true and not very useful. We're more interested in why an instruction the system demonstrably understood failed to bind it.

And it did understand. Any competent language model can define "must," tell it apart from "should," explain why the difference...

Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE

Read more