Stop Testing AI Agents Like Normal Functions
Every developer building AI agents eventually hits the same wall. A test passes, then fails because the response changed from “I'll help you book that flight” to “Happy to assist with your travel plans.” Both are acceptable, but the exact-string assertion is red.
Fuzzy matching can become too loose. A stricter prompt makes the agent robotic. Temperature zero reduces variation but does not turn a remote model, toolchain, and changing environment into a pure function. Mocking the model makes tests fast, but it no longer tests the model's behavior.
The answer is not to stop testing. It is to test the contracts around the probabilistic component: schemas, proposed actions, executed tools, authorization, state transitions, outcomes, and regressions.
The First Test Often Targets the Wrong Contract
This test is understandable but brittle:
test("agent responds helpfully", async () => { const response = await travelAgent.run("Find hotels in Paris"); expect(response.text).toBe( "I'll help you...
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE
