How to Make the Semantic Layer Work for Analytics Agents

The Semantic Layer Challenge

The semantic layer is often promoted as essential for reliable analytics agents, but initial testing revealed a critical flaw: “the semantic layer was so restrictive that the agent ended up answering no questions.” Through iterative testing using dbt MetricFlow and the nao evaluation framework, the author achieved a dramatic improvement from 4% to 82% reliability.

4 Steps to Implement a Semantic Layer Agent

Step 0: Re-establish the Baseline

Starting with a schema, data samples, and rules.md with a Claude Code-generated semantic layer yielded poor results. After improving the underlying data model, the new baseline showed 13% correct answers, 61% unanswered, and 26% wrong.

Step 1: Improve Semantic Layer with dbt Skills

Installing dbt skills to create a better semantic layer improved results to 27% correct, 17% unanswered, and 57% wrong. Deep analysis revealed two failure categories:

Missing metrics and dimensions in the semantic layer
Metrics too complex for the semantic layer without data model changes

Step 2: Add Missing Dimensions and Metrics

Adding missing elements without changing the data model brought performance to 47% correct, 5% unanswered, and 48% wrong. Analysis identified persistent issues:

Date filtering ambiguity: Questions about “last X weeks” lack clarity on week boundaries
Data quality gaps: Null values weren’t being filtered appropriately
Missing business logic: Relationships between fields weren’t documented (e.g., pricing only exists for paying users)
Join problems: Duplicate primary keys caused incorrect joins
Out-of-scope queries: Some questions required sub-CTEs or pre-aggregations

Step 3: Improve Agent Rules

Enhancing the semantic layer and rules.md addressed identified issues:

Clarified date filtering approaches for the semantic layer
Documented nullable dimensions
Added business context to field descriptions
Reviewed entity keys to fix join logic
Enabled natural language querying as a fallback when semantic layer proved insufficient

Results: 76% correct, 7% unanswered, 18% false

Step 4: Enrich Data Model

Final improvements required data model changes. Example: computing churned MRR at subscription granularity required pre-computing the metric at row level before aggregation.

Final Results: 82% correct, 11% unanswered, 8% wrong

Key Learnings: 7 Elements for Semantic Layer Reliability

Use dbt semantic layer skills to construct the semantic layer
Employ dbt natural language querying skills to let the agent choose between semantic layer and SQL tools
Build exhaustive data models with pre-computed metrics and dimensions
Manually review semantic layer entities and their keys to prevent bad joins
Enrich semantic layer documentation with business context in descriptions
Add agent rules for date filtering, null handling, and ambiguous query resolution
Include a context layer alongside the semantic layer for independent metric and table access

The Honest Verdict: Is the Semantic Layer Worth It?

The semantic layer addresses only half the agentic analytics challenge. The process involves two distinct steps:

Metric/entity selection (non-deterministic)
Query generation (deterministic with semantic layer; non-deterministic with text-to-SQL)

“The semantic layer reduces hallucinations, but doesn’t remove them entirely.” Context layers remain essential, adding business definitions and querying rules.

Notably, an agent without the semantic layer achieved 86% reliability on the same dataset. Semantic layers show greater value with larger, more complex data models. However, they come with tradeoffs:

Higher costs: dbt MCP integration requires numerous tool calls and token consumption
Slower performance: Increased duration compared to non-semantic baseline
Model constraints: Reliability dropped to 57% when tested with Claude Haiku

The final optimization balances question coverage, accuracy rate, speed, and cost.

Recommended Next Steps

Use dbt semantic layer skills to create the semantic layer
Create a nao project with unit tests
Evaluate agent performance against current semantic layer
Categorize failures as data issues, documentation gaps, test problems, or model issues
Fix one category at a time and re-test between iterations