Testing and evaluating agents
You test an analytics agent by running it against questions whose answers you already know and checking that it gets them right. This happens before you deploy, not after. An untested agent looks fine in a demo and fails quietly in front of real users, where a wrong number is hard to catch.
Testing is what separates an agent you can trust from one you hope works. The whole point of the agentic loop is reliability, and you only know it is reliable if you measure it.
Why you test before deploying
A business user cannot tell a wrong answer from a right one. The number comes back clean either way. So the check has to happen before the user sees it, while you still have the known-correct answer to compare against.
Testing also protects you from change. Every edit to a metric, a rule, or the schema can break an answer that used to work. Without tests you find out when someone complains.
Unit tests for accuracy
In nao you write unit tests with nao test. Each test is a question paired with the answer you know is correct. The agent runs the question, and the test compares its result against the expected one.
- Known answer. “Revenue for Q1 was 1.2M.” The test checks the agent returns that.
- Known shape. “Top five products by sales.” The test checks the list and the order.
- Known rule. “Exclude test accounts.” The test checks they are gone.
A test suite turns reliability from a feeling into a number you can watch.
Evaluating and catching regressions
Run the suite whenever the context changes. If you redefine active user and three tests start failing, you caught the break before it shipped. That is a regression test doing its job.
Over time the suite grows. Every wrong answer you find becomes a new test, so the same mistake cannot come back. This is how an agent gets more trustworthy instead of drifting.
Testing is not a one-time gate. It runs again every time you update the context, which is part of keeping the agent reliable. With a passing suite, you are ready to deploy to your team.