How to evaluate an analytics agent
Before you let an analytics agent answer questions for your company, you need to know how often it is right. Evaluating an analytics agent means building a set of real questions with known correct answers, running the agent against them, and measuring how many it gets right. That number is the difference between a tool people trust and a tool that quietly misleads them.
This is the step most teams skip, and it is the one that matters most. An agent that is right 70 percent of the time feels impressive in a demo and is dangerous in production, because you cannot tell which 30 percent are wrong.
Why evals matter more for analytics
Every LLM product needs evaluation. Hamel Husain has argued this repeatedly. The teams whose AI products fail almost always share one root cause, which is no real evaluation system.
Analytics makes it worse for one reason. A wrong answer looks exactly like a right one. When a coding agent fails, the code usually breaks. When an analytics agent fails, it returns a number formatted like any other number. There is no error, no red flag, just a decision made on bad data. So you cannot rely on catching failures in use. You have to measure accuracy up front.
Build a test set of real questions
The core of an eval is a set of questions paired with their correct answers. Not made-up questions. The actual questions your team asks, with answers a human analyst has verified.
Cover the range. Include simple lookups, questions that need a join, questions about defined metrics, and the ambiguous ones where the right move is to ask for clarification. Include the questions that have burned you before. The point is to represent what the agent will actually face, not to make it look good.
Public benchmarks help as a reference here. BIRD and Spider 2.0 are large text-to-SQL test sets built on real databases, and they are useful for seeing how hard the general problem is. But your own questions on your own data are what tell you if the agent works for you.
Measure the right thing
Execution accuracy is the number that matters most. Run the agent’s query and the known-correct query, and check whether the results match. This catches the silent failures that “does the SQL look reasonable” misses.
For questions without a single clean answer, or when you want to grade the explanation and not just the number, you can use another model as a judge. Hamel Husain has a full guide to LLM-as-a-judge, and the key caution is that a judge is only useful once you have checked that it agrees with human judgment. Eugene Yan’s notes on task-specific evals are a good reference for which metrics actually work for which task.
Track a couple of things beyond accuracy. How often the agent correctly refuses or asks for clarification instead of guessing, and how well accuracy holds up as the questions get harder.
Make it a regression test
An eval is not a one-time report. It is a test suite you run every time something changes, whether that is a new model, a new prompt, or a change to the semantic layer or the context. Accuracy can drop from a change that looked unrelated, and without a suite you will not notice until someone acts on a wrong number.
The teams that do this well treat it like software testing. Some analytics agents build it in. Open-source ones like nao ship a test command so you can run your question set against the agent before you deploy and catch regressions after. The learn path walks through setting this up in practice.
The short checklist
- A set of real questions with human-verified answers, covering easy, hard, and ambiguous cases.
- Execution accuracy as the primary metric, checked by comparing results, not just reading the SQL.
- An LLM judge for open-ended answers, validated against human grading first.
- The whole thing wired as a regression test you run on every change.
Measure first. An agent you have not evaluated is an agent you cannot trust, no matter how good the demo looked. If you are comparing tools, the compare page flags which ones support evaluation out of the box.
How do you measure the accuracy of an analytics agent?
Run it against a set of questions whose correct answers you already know, then compare the agent’s query result to the known-correct result. This is called execution accuracy, and it catches silent failures that reading the SQL would miss. Cover easy, hard, and ambiguous questions on your own data.
What is LLM-as-a-judge in analytics evaluation?
LLM-as-a-judge uses a second model to grade an agent’s answer, which is useful for open-ended questions or for scoring the explanation rather than the number. It only works once you have confirmed the judge agrees with a human on a sample, otherwise you are trusting one unchecked model to grade another.
How often should you re-run analytics agent evals?
Every time something changes, whether a new model version, a prompt change, or an update to the semantic layer or context. Accuracy can regress from a change that looked unrelated, so the eval should run like a regression test, not as a one-time check.