How to Build a Context Stack for Agentic Analytics

Introduction

Most teams attempting to deploy analytics agents encounter the same obstacle: “the model is not the real bottleneck, context quality is.” Rather than relying solely on prompts, reliable agentic analytics requires a structured context stack integrated with existing data infrastructure, dbt projects, and chat-with-data workflows.

The nao open source framework provides scaffolding to implement this approach directly from the public repository.

Why Context Beats Model Upgrades

When analytics agents exhibit inconsistent behavior, the underlying causes typically include:

Absence of definitive metric definitions
Inconsistent documentation across platforms
Excessive raw context supplied to the model
Missing evaluation processes following context modifications

Context engineering offers a repeatable methodology to enhance answer quality beyond random prompt adjustments.

7 Steps to Build a Context Stack

1. Define Reliability KPIs Before You Ship

Track these four metrics from launch:

Answer rate: percentage of questions the agent successfully answers
Accuracy: percentage of correct answers provided
Cost per question: combined token and tool expenses
Response time: complete end-to-end latency

Without measuring these metrics, improvement becomes impossible.

The nao testing workflow includes KPI tracking through nao test commands and test result exploration via nao test server.

2. Create a Context Source-of-Truth Policy

Establish a documented priority order for definition sources:

dbt models and documentation
Versioned rules (RULES.md)
Curated semantic definitions
Human-readable runbooks

This hierarchical approach prevents the agent from selecting conflicting definitions and prevents drift.

nao implements this through versioned instructions in RULES.md and source configuration in nao_config.yaml, maintaining inspectable and Git-reviewable governance.

3. Start with a Minimum Viable Context Set

Avoid inserting every available document into context. Begin with:

Warehouse schema metadata
dbt model descriptions and lineage
Core business metric definitions
A concise rules file covering date logic and entity conventions

This foundation typically provides sufficient utility while controlling costs.

In nao, this minimum set corresponds to repository primitives: synchronized database context, documentation folders, and agent configuration files.

4. Format Context for Machine Readability

Construct context that is modular and explicit:

Short sections featuring stable headings
Single definition per metric
Explicit date-window rules
Clear table ownership documentation

This structure reduces ambiguity and accelerates model table selection.

nao’s context builder generates machine-readable structures, rendering context files from templates during synchronization.

5. Add Unit Tests for Business Questions

Develop a test suite containing real prompts paired with expected SQL queries or outputs. Execute tests after every context modification.

An effective initial suite comprises 20 to 40 questions covering revenue, churn, signup, retention, and weekly reporting scenarios.

Reference nao’s public test examples, like example/tests/total_revenue.yml, to establish patterns before scaling to custom benchmarks.

6. Set a Weekly Context Refresh Workflow

Treat context as production-grade infrastructure:

Retrieve new dbt documentation and schema updates
Eliminate outdated definitions
Examine user feedback indicators
Incorporate unsuccessful questions into test suites

This recurring cycle transforms prototypes into production analytics systems.

nao supports this through nao sync for periodic context refresh from configured sources, followed by nao test for validation.

7. Connect Monitoring Back to Context Changes

When reliability decreases, investigate context modifications rather than only examining model logs:

Newly introduced model or table names
Metric renamings
Interrupted joins
Expired rules

A straightforward changelog connected to test executions typically enables rapid diagnosis of regressions.

Since nao maintains context as files and tests as YAML, reliability issues trace back to concrete Git diffs instead of opaque vendor processes.

Common Architecture That Works in Practice

Successful implementations typically employ this structure:

dbt: primary warehouse documentation and transformation layer
Versioned context folder: containing rules, glossary, and examples
Evaluation runner: for prompt test execution
Lightweight monitoring dashboard: for oversight

This pattern preserves interoperability and avoids vendor dependency, enabling data teams to maintain quality ownership while users interact through familiar interfaces.

Reference the nao project structure for concrete implementation: example/nao_config.yaml, example/RULES.md, example/tests/, and example/agent/mcps/mcp.json.

Final Takeaway

“The teams that win with analytics agents are not the ones with the flashiest UI. They are the teams that run context engineering as an operating system: define truth, measure outcomes, iterate every week.”

Success requires designing your context stack first. Everything else becomes simpler afterward.