Benchmarking Cheap LLMs for Production Agent Traces

We run one LLM call on every agent trace we ingest: it reduces the trace to a short, searchable digest. Because it runs on every trace from every customer, it is the most frequent LLM call we make, and its cost is the one that grows fastest with our usage.

So we tested whether a cheaper model could do it. Sonnet 4.6 writes that digest today at about $3.45 per 1,000 traces, and plenty of models cost a fraction of that. We put four of them against Sonnet on a frozen set of 50 traces. The one we shipped runs at about a thirtieth of the cost with search returning the same traces, and the two that looked cheapest on the price sheet both failed.

The harder part was checking it. A cheaper model can write summaries that read perfectly and still change what search returns, so we graded each...

Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE