The price is wrong: AI cost calculation has to consider task completion rates, not just token costs
AI and ML
Cheap can be expensive
When it comes to AI services, you don't necessarily get what you pay for.
It turns out that AI models with expensive tokens may cost less than models with cheap tokens for particular tasks. And the tooling attached to those models can have a significant effect on cost and output quality.
Databricks, which sells data analytics software and services, recently devised an internal coding benchmark to assess the tradeoff between price and performance using various AI models.
Matei Zaharia, CTO of Databricks and associate professor of computer science at UC Berkeley, said the company undertook the evaluation because models are often tuned to existing benchmark tests like SWE-Bench – which is "broken," according to OpenAI.
Databricks devised its benchmark using real engineering tasks performed by its staff to assess how AI agents perform. Zaharia said while the results reflect the company's internal...
Copyright of this story solely belongs to theregister.com. To see the full text click HERE
