How to Evaluate an Analytics Agent: A Practical Guide with nao test

The Core Idea: Unit Tests for SQL

The framework operates on a straightforward principle. Compose questions your users genuinely ask, match each with the SQL query yielding the correct answer, and allow the system to execute the agent across all of them.

Each test contains three elements:

• name, a descriptive identifier
• prompt, the natural language question
• sql, the query that produces the ground truth

The framework transmits the prompt to your running agent, records the answer, runs your SQL to obtain expected data, and performs a comparison. A test succeeds only when the data aligns precisely, not when the SQL resembles the expected query.

This methodology identifies the most critical failure pattern in production analytics agents: “SQL that is syntactically valid but semantically wrong.”

Setting Up Your Test Suite

Establish a tests/ folder in your nao project root:

your-project/
├── nao_config.yaml
├── RULES.md
├── tests/
│ ├── total_revenue.yml
│ ├── weekly_signups.yml
│ └── outputs/   (auto-generated results)

Create your test files. Begin with the 10 to 20 most-frequently-asked questions from your team: funnel, retention, revenue, churn. Incorporate questions with documented correct answers to establish legitimate ground truth.

name: total_revenue
prompt: What is the total revenue from all orders?
sql: |
  SELECT SUM(amount) as total_revenue
  FROM orders

name: signups_weekly
prompt: How many signups did we have per week in the last 4 weeks?
sql: |
  select week, sum(n_new_users) as n_signups
  from prod_silver.fct_users_activity_weekly
  where week >= date_trunc(current_date - interval 4 week, isoweek)
  and week < date_trunc(current_date, isoweek)
  group by week
  order by week

Include edge cases as well: conflicting metric names, unclear date windows, multi-table joins. “Happy-path accuracy does not predict production reliability. Edge case accuracy does.”

Running the Evaluation

Initiate your nao chat server, then execute:

nao test

The command identifies all .yml files in your tests/ folder, processes each through your agent (defaulting to openai:gpt-4.1), and shows a summary table displaying pass/fail status, token usage, cost, execution time, and tool call count.

For faster execution with parallel threads:

nao test -t 4

To evaluate a specific model:

nao test -m openai:gpt-4.1

Results are automatically recorded to tests/outputs/results_TIMESTAMP.json, containing full conversation history, tool calls, actual vs expected data, and a detailed diff when failures occur.

How It Works Under the Hood

For every test, the framework:

1. Delivers the prompt to your agent and executes it normally. The agent may execute SQL, search context, use tools
2. Collects the entire conversation history and response
3. Extracts the agent’s final answer as structured data using a secondary LLM call
4. Runs your expected SQL against the warehouse to obtain ground truth
5. Compares the two datasets: exact match first, then approximate match with numeric tolerance for floating-point differences
6. If both comparisons fail, generates a detailed diff showing exactly where values diverge

The comparison enforces strict criteria intentionally. “Row count mismatch = fail. Semantic equivalence is not enough. The data has to match.”

Visualizing Results with nao test server

Once you have executed nao test at least once, open the visual interface:

nao test server

This launches a dashboard at http://localhost:8765 featuring:

• Summary cards, pass rate, total tests, token cost, total duration
• Results table, all test runs with status, metrics, and inline details
• Detail view, click any test to see the full response, tool calls, actual vs expected data, and a diff for failures

The visual interface facilitates identifying patterns: which question types are failing, where tool call counts spike, which context changes caused regressions.

What Good Evaluation Looks Like in Practice

The initial execution of nao test on a newly-created context setup will not achieve 100%. That represents expected and valuable feedback. The number demonstrates where your context needs enhancement.

The workflow proceeds as:

1. Execute nao test, identify which questions fail
2. Review the detail view in nao test server, connect the failure to a specific context gap (wrong table selected, missing metric definition, bad join key)
3. Revise the context: modify RULES.md, contribute a column description, clarify a join rule
4. Re-run nao test, verify the fix did not introduce a regression

This cycle runs every time context changes. The complete test suite takes under a minute for 40 questions with -t 4. Execution speed permits running on every context commit.

For context: “our best-performing context configuration, schema + data sampling + RULES.md, restricted to the silver layer, reached 45% reliability on our 40-question test suite on the first run.” Context iteration from there improves that number. Review the context impact study for the complete breakdown of which context elements actually affect performance.

The Self-Learning Loop: From Feedback to Better Context

Unit tests reveal whether your agent performs correctly on recognized questions. When you launch to users, you require a second signal: what are real users asking, and when does the agent fail?

nao integrates this loop with a built-in feedback and monitoring system.

In the chat interface, users can rate any answer with a thumbs up or thumbs down. They can optionally add a reason. This establishes a lightweight feedback signal directly connected to actual production queries, not your test suite.

In the admin monitoring dashboard, you examine all feedbacks consolidated:

• Which answers received negative ratings
• What the user asked and what the agent responded
• Who flagged it and when

This functions as the memory component. Each negative feedback represents a signal that something in your context is missing or wrong, a metric definition, a join rule, an undocumented edge case.

The improvement cycle:

1. Execute nao test, fix context gaps on your recognized question set
2. Launch to users, gather real feedback from the chat
3. Examine negative feedback in the monitoring dashboard, identify fresh gaps
4. Modify RULES.md or schema context to address them
5. Introduce the failing question to your tests/ folder as a fresh unit test
6. Re-run nao test, verify the fix did not regress anything

Gradually, your test suite expands to reflect the questions your users genuinely ask. Your context becomes progressively more accurate. The agent self-improves, not through model fine-tuning, but through “structured context iteration driven by real usage.”

This illustrates why evaluation and monitoring constitute a unified loop, executing at two speeds: automated unit tests for rapid iteration, user feedback for continuous production enhancement.

Best Practices

Start with critical queries. Test the questions most significant to your business users, the ones that would produce real consequences if incorrect. Incorporate edge cases after covering the high-value baseline.

Keep tests focused. Each test should verify one behavior. A test inquiring about MRR by segment by channel by month becomes difficult to debug when it fails. Divide it into smaller tests.

Avoid leakage. Do not integrate exact answers or overly particular details in your RULES.md that permit the agent to pattern-match rather than reason. The test gauges whether your context enables correct reasoning, not whether the agent memorized your test cases.

Commit your tests folder to git. Your test suite constitutes a versioned record of what “correct” means for your data model. It should exist alongside your context files, not in a separate spreadsheet.

Run tests after every context change. Agent reliability can deteriorate when you modify RULES.md, introduce new tables, or update metric definitions. Automated evaluation identifies this before users encounter it.

Related Reading

• What Context Has the Most Impact on Analytics Agent Performance?, which context pieces actually move accuracy in our 40-question benchmark
• What Is the Impact of a Semantic Layer?, does adding MetricFlow improve reliability? We tested it.

Full documentation for nao test and nao test server is at docs.getnao.io/nao-agent/context-engineering/evaluation.