4 Steps to Improve Your Analytics Agent Reliability from 45% to 86%

Introduction

In the first context engineering study, the best setup achieved 45% reliability on 40 unit tests, a significant improvement over the 17% baseline, but insufficient for business users to “chat with data” confidently.

This second study focused on understanding why the analytics agent was still failing, then fixing the context incrementally and measuring each change’s impact. The result: reliability improved from 45% to 86%.

For teams building agentic analytics on dbt and modern data stacks, the practical lesson is clear: most gains do not come from adding orchestration or model cleverness. They come from better context engineering and a cleaner data model, even in open source setups.

Step 1. Diagnose why your analytics agent is failing

Before changing anything, failure distribution across the 40 tests was analyzed in top-performing context configurations.

The pattern was not random:

• 13 tests never returned a correct answer
• 12 tests were correct more than 80% of the time
• The remaining tests were inconsistent

This indicated the agent was not simply hallucinating unpredictably. Certain questions systematically broke the system.

For the 28 tests with less than 80% success, the full agentic loop was reviewed for each test:

• What mistake the agent made
• What context it retrieved
• What assumption it made

Every failure was categorized into three buckets:

• 18 data model errors
• 6 date selection errors
• 4 test issues

What these failures actually meant

Most failures stemmed from data-model quality issues. Some fields were unclear, deprecated, or missing entirely. The agent frequently had to guess which table was the source of truth for a metric. In other cases, the right model existed but lacked sufficient documentation.

Date selection was the second major issue. These failures were often not obviously incorrect, but they produced inconsistent outputs. For example, “last 8 weeks” could be interpreted as 56 rolling days, eight full ISO weeks, or a range including the current incomplete week. Such instability is enough to break user trust.

The final category involved the tests themselves. One expected answer was wrong, and several tests did not specify formatting clearly enough for fair comparison.

Step 2. Re-establish the baseline and fix test issues first

Before making context changes, the best setup from the first study was re-run on current data.

The original best context remained simple:

• Table schema
• 10-row table previews
• rules.md

That rerun moved reliability from 45% to 49%.

Then test issues were fixed:

• Corrected wrong expected outputs
• Clarified required output format where comparisons were too loose

This only moved reliability from 49% to 51%, but it mattered. If the benchmark is noisy, the optimization loop cannot be trusted.

Step 3. Improve date selection rules to remove instability

The next improvement involved rewriting date-selection rules.

Rules already existed, but they remained too ambiguous. For example:

"Last X weeks/days": Use >= date_trunc(current_date, isoweek) - interval X week

When failures were traced, the agent interpreted this in several ways:

• Using TIMESTAMP() instead of DATE()
• Subtracting weeks from today instead of aligning to ISO week boundaries
• Including the current incomplete week
• Converting weeks into rolling days

Each interpretation was plausible, but they produced different numbers.

The rules were rewritten with explicit DO and DON’T guidance.

Example rule for “last X weeks”

When a user asks about “last X weeks”, use full ISO weeks from Monday to Sunday and exclude the current incomplete week unless the user explicitly asks otherwise.

date(created_at) >= date_trunc(current_date, isoweek) - interval 3 week and date(created_at) < date_trunc(current_date, isoweek)

Do not:

• Use timestamp comparisons when the metric should be date-based
• Interpret 3 weeks as 21 rolling days
• Include the current partial week by default

This single class of rule changes moved reliability from 51% to 60%.

This reminder is useful for any analytics agent workflow: some failures are not about wrong SQL, but about unstable conventions. If users ask the same question on two different days and get different logic, they stop trusting the system.

Step 4. Add missing documentation in dbt and rules

After addressing dates, the focus shifted to missing documentation.

Missing definitions were added to dbt docs and the most important business definitions were copied into rules.md.

This moved reliability from 60% to 65%.

This step did not transform the benchmark by itself, but it reduced the guessing the agent had to do. If business logic only exists in people’s heads, the model will improvise. If the logic is written down in the repository, the model can follow it.

Step 5. Improve the data model and clarify source of truth

The biggest gain came from improving the data model itself.

Reliability moved from 65% to 86%.

Several kinds of fixes were made:

• Added missing fields the agent had previously been forced to infer
• Materialized useful booleans and computed columns instead of expecting the agent to reconstruct them
• Clarified which table was the source of truth for each key metric
• Split ambiguous business concepts into separate metrics when one name actually covered multiple realities

One recurring issue was that a metric like “paying users” could mean either paid licenses or users with an attributed paid license. These are not the same thing. Once these definitions were split and documented clearly, the agent stopped making hidden assumptions.

Explicit rules were also added:

• Source-of-truth mappings for key metrics
• Definitions for ambiguous phrases such as “our users”
• Instructions to state which user definition was chosen in the final answer

For example, “our users” could reasonably mean:

• All signed-up users
• Active users in a time window
• Paying users

If the term is ambiguous, the agent should infer the most likely meaning from context and explicitly state which definition it used.

The study’s main conclusion became obvious at this point: the largest reliability gains did not come from fancy context formats. They came from cleaning up the underlying analytics model.

What the remaining 14% of errors actually mean

The final 14% of errors were not mostly nonsense outputs. They were usually interpretation choices that a human would resolve with a follow-up question.

For example:

• Does ”% of users with more than one warehouse” mean out of all users, or only out of users who already have at least one warehouse?

Pushing this benchmark all the way to 100% would likely mean overfitting to one expected SQL answer rather than improving real production usefulness.

At 86%, the agent was already much closer to something that could be rolled out in a controlled scope, then improved from live feedback.

Key learnings for context engineering

If the study is reduced to five practical takeaways, they are these:

1. Clean the data model until the ambiguity is gone

If the agent does not know which table to pick, there is a good chance the data team does not either.

2. Do not make the agent guess fields you could materialize

Humans prefer tidy models. Agents often perform better when the model includes the exact boolean or computed field they need instead of forcing them to derive it every time.

3. Remove instability factors, not just obvious errors

Date-range logic and source-of-truth selection can both produce inconsistent answers even when each individual interpretation looks defensible.

4. Anticipate user imprecision

Terms like “our users”, “currently”, or “active” need explicit rules or follow-up questions.

5. Do not overthink context format too early

You do not need a huge semantic layer strategy on day one to get meaningful gains. In this study, the biggest improvements came from better metadata, better documentation, and better rules written in straightforward markdown files.

This work fits naturally into an open source analytics workflow. Context can be versioned, changes inspected, benchmarks measured, and the system improved step by step on top of existing dbt projects and data stacks.

Which tools should you choose to improve analytics-agent reliability?

If the goal is to improve reliability rather than just produce a good demo, the stack should optimize for inspection and iteration:

• dbt for transformation logic and documentation
• Versioned rules and metadata files
• A measurable evaluation loop
• An open source framework where context changes are visible

For teams serious about reliability, nao is a strong option for both creating and testing an open source analytics workflow. The important part is not just that it helps ship an agent, but that it gives a structured way to measure whether the agent is actually getting better.