How to do context engineering for analytics agents

What actually moves agent reliability

The author conducted a first study testing 30 different context setups against 40 text-to-SQL tests. The best setup reached 45% reliability. Nothing exotic. Just three pieces of context: data schema, a sample of rows per table, and a rules.md file with business logic and edge cases.

Elements that did not improve performance included:

• A full dbt repository in context (reduced performance due to noise)
• MetricFlow YAML as a semantic layer shortcut (no improvement versus rules.md)
• Metrics-store-style semantic layers (decreased coverage)

The rules.md file emerged as the primary performance driver, nearly matching results from comprehensive schema and profiling combinations.

A second study improved the baseline from 45% to 86% through failure analysis of the 28 failing tests. These failures broke down as:

• 18 data model errors
• 6 date selection errors
• 4 test errors

The improvement sequence involved fixing test errors first, then adding explicit date selection rules with examples, followed by data model cleanup including computed fields, metric documentation, and term clarification.

Key insight: context engineering is mostly data engineering. Most agent failures are data model failures in disguise.

Should you add a semantic layer?

When testing a dbt MetricFlow semantic layer, adding it reduced hallucinations but also reduced answers to almost zero. It was 4x more costly in tool calls and 3x slower, and the semantic YAML added no performance versus a plain rules.md.

In the third study, forcing the agent to use only the semantic layer yielded 82% reliability through:

1. Using the dbt semantic layer skill to generate it
2. Employing natural language querying skills for agent choice between semantic layer and raw SQL
3. Making the semantic layer exhaustive
4. Manual entity and key review
5. Enriching descriptions with business context
6. Adding rules.md rules for date filtering, null handling, and ambiguous terms
7. Adding a nao context layer for discovery independence

The verdict: the semantic layer reduces hallucinations, it does not remove them. The agent still has to pick the right metric among many. At 12 tables with clean models, both approaches reached similar reliability levels. Semantic-layer agents proved more expensive and slower. A complexity threshold around hundreds of tables with heavy joins may justify the investment.

Recommendation: skip semantic layers initially. Invest in rules.md and clean data models first. Only pursue semantic layers if managing 150+ tables across multiple domains with governance challenges.

How to set up a testing and monitoring framework

Success requires measurement. The framework tracks five metrics:

• Coverage: percentage of questions the agent attempts to answer
• Reliability: percentage of correct answers
• Cost: tokens spent per answer
• Speed: time to first useful response
• Data scanned: volume queried against the warehouse

Build a unit test set of 20 to 50 real business questions paired with expected SQL, stored as YAML. Include KPI lookups, cohorts, error analysis, and distributions with single-table and multi-join queries.

Run evaluation after every context change, treating rules.md edits and data model changes like model releases. Categorize failures by root cause (data model, date scope, business ambiguity, test error, out-of-scope) rather than fixing randomly. This classification reveals investment priorities.

Monitor production by logging user questions, agent answers, and flagging runs for human review. Feed confirmed failures back into the test set to create a continuous improvement loop.

Where to start

For data teams beginning today:

1. Write a minimal rules.md with business definitions, ambiguous-term rules, and date logic
2. Add schema and small data samples to agent context
3. Set up 20 unit tests based on real user questions
4. Run an evaluation baseline
5. Perform failure analysis, fixing data model issues first, then rules
6. Skip the semantic layer unless managing a large, messy model

Final takeaway

Context engineering is not a tooling question. It is a discipline: clean data models, explicit rules, versioned context, and a test loop you run every time something changes. Teams treating context as infrastructure will ship reliable agents. Those hoping semantic layers solve reliability issues will not.