What 46,000 Production Queries Taught Us About LLM-Generated SQL

https://hackernoon.imgix.net/images/2jqChkrv03exBUgkLrDzIbfM99q2-n2822ku.jpeg

We run a financial data server that sits on top of a 31-million-row DuckDB database. Users connect an LLM (usually Claude), ask questions in plain English, and the model generates SQL against our schema.

Over six months we logged every query: the user's natural language prompt, the SQL the LLM produced, whether it succeeded, how long it took, and how many rows came back. 46k queries total.

Here is what we found.

The database

The database covers 64 years of US stock market data across 12 tables: daily prices and volume (31M rows), quarterly and yearly financials, daily valuations, 56 technical indicators, SEC filing metadata, insider transactions, analyst estimates, and company reference data. About 9,971 securities (stocks and ETFs) on NYSE and NASDAQ.

Users connect to it via MCP (Model Context Protocol), the open standard that lets LLMs call external tools. When a user asks "show me US stocks with...

Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE