Publications

Writing worth your time

Everything worth reading on agentic analytics in one feed. Our guides and benchmarks mixed with the best writing from across the data world. Ours are marked nao.

Topics

Guides· Jul 2026Agentic Analytics
Is read-only MCP access to your database safe?Read-only database access sounds safe for an AI agent, but a real SQL-injection flaw in a popular MCP server bypassed it. Why it is not enough, and what is.
Guides· Jul 2026Agentic Analytics
What is agentic analytics?Agentic analytics is AI agents that answer data questions in plain English by writing their own queries. How it works, why it is hard, what makes one reliable.
Guides· Jul 2026Agentic Analytics
Text-to-SQL vs the semantic layerText-to-SQL writes queries on the fly, a semantic layer serves predefined metrics. How they differ, what the benchmarks show, and when to use each in an agent.
Guides· Jul 2026Agentic Analytics
How to evaluate an analytics agentHow to evaluate an analytics agent before you trust it. Build a test set, measure execution accuracy, use an LLM judge, and run evals as regression tests.
Guides· Jul 2026Agentic Analytics
Why text-to-SQL fails on real dataText-to-SQL works in demos but fails silently on real warehouses. Why it happens, from ambiguous metrics to wrong joins and missing context, and what fixes it.
Guides· Jul 2026Agentic Analytics
Open-source analytics agents, explainedAn open-source analytics agent is one you can read, self-host, and control. Why that matters for data teams, the tradeoffs against closed SaaS, and what to look for.
Guides· Jul 2026Agentic Analytics
How to build an analytics agentA practical guide to building an analytics agent. Connect your warehouse, engineer the context, give it a correction loop, evaluate accuracy, and deploy.
Guides· Jun 2026Agentic Analytics
Agentic analytics use casesWhere agentic analytics actually helps, from revenue and finance questions to product analytics, ops, marketing, and giving execs self-serve answers without SQL.
Guides· Jun 2026Agentic Analytics
What is an AI data analyst?An AI data analyst answers data questions in plain English by writing its own queries. What it can do, where it falls short, and how it compares to a human.
Guides· Jun 2026Agentic Analytics
Conversational analytics, explainedConversational analytics lets anyone query data by asking in plain English. How it works, how it differs from dashboards, and what makes it reliable.
Guides· Jun 2026Agentic Analytics
Is it safe to give an AI agent access to your data?The real risks of giving an analytics agent your data, from exposure to wrong answers, and how to run one safely with governance and self-hosting.
Guides· Jun 2026Agentic Analytics
How to run analytics from Slack or TeamsPut an analytics agent in Slack or Teams so anyone can ask data questions where they work. How it works, why chat fits, and what to watch for.
Guides· Jun 2026Agentic Analytics
Why MCP alone is not enough for AI analyticsMCP connects an AI agent to your data, but it does not make the answers correct. Why analytics agents need a semantic layer on top of MCP, and what Gartner says.
Guides· Jun 2026Agentic Analytics
How to test a text-to-SQL agent in productionA hands-on guide to testing a text-to-SQL agent. Build a question set with known answers, measure execution accuracy, and catch silent drift in production.
Guides· Jun 2026Agentic Analytics
How to self-host a semantic layer for AI agentsWhy self-host a semantic layer for AI agents, the dbt Cloud constraint, the open-source options like MetricFlow and Cube, and how it fits an analytics agent.
Guides· Jun 2026nao
How agentic analytics worksAn agentic analytics system answers a question by turning it into a plan, writing SQL, running it, checking the result, and correcting itself until the answer holds up.
Guides· Jun 2026nao
What is agentic analytics?Agentic analytics is when an AI agent answers a data question by planning a query, running it on your warehouse, checking the result, and replying in plain language.
Opinion· Jun 2026Seattle Data Guy
In 2026 The Data Fundamentals Matter More Than EverA working data consultant argues that messy foundations and weak data quality, not missing AI, are what actually block companies from getting value from agents.
Guides· Jun 2026ByteByteGo Newsletter
How OpenAI Built Its Data AgentBreaks down OpenAI's internal data agent serving ~4,000 users over 90,000 datasets, showing reliability comes from layered prepared context and a curated toolset rather than model sophistication.
Research· May 2026nao
LangChain vs Wren AI vs nao: which open-source tool for agentic analytics?Three open-source projects for agentic analytics tested on the same BigQuery question. Setup, accuracy, context approach, and which one to pick.
Guides· May 2026OpenAI Cookbook
Building Reliable Agents with Memory and CompactionA hands-on guide showing two reliability primitives (compaction for long conversations and memory for reusing lessons across runs).
Opinion· Apr 2026Omni Blog
Why text-to-SQL failsArgues text-to-SQL fails silently and confidently on enterprise data through metric misinterpretation, non-deterministic generation and wrong join paths, and that a governed semantic layer is the fix.
Guides· Apr 2026nao
How to Make the Semantic Layer Work for Analytics AgentsHow to go from a semantic layer answering no questions to 82% reliability across a wide business range, in 4 steps with real benchmark data.
Research· Apr 2026dbt Developer Blog
Semantic Layer vs. Text-to-SQL: 2026 Benchmark UpdateA benchmark finding the dbt Semantic Layer reached 98-100% accuracy versus 84-90% for text-to-SQL on a 15-table dataset, with deterministic query generation preventing silently incorrect answers.
Guides· Apr 2026nao
How to Do Data Modeling for AI Agents: 8 Practical RulesA practical guide to data modeling for AI agents, with clear rules to optimize for precision, reliability, and better chat-with-data performance.
Guides· Mar 2026nao
How to Build a Context Stack for Agentic AnalyticsA practical 7-step guide for data teams to build a context stack that improves analytics agent reliability, speed, and cost control.
Guides· Mar 2026nao
How to do context engineering for analytics agentsA summary of three context engineering studies on analytics agents. What actually moves reliability, whether a semantic layer is worth it, and how to set up a testing and monitoring framework.
Guides· Mar 2026Data Analysis Journal
Before You Build an Analytics AgentReframes 'AI-readiness' as an ongoing process and lays out the data-prep, guardrails and evaluation groundwork teams need before deploying an analytics agent.
Research· Mar 2026nao
What's the Best Analytics Agent for Your Data Team?A practical guide to choose the best analytics agent option for your data team, comparing buy, setup, build, and open-source approaches.
Guides· Mar 2026nao
How to Set Up an AI Analytics Slack Bot with an Open Source FrameworkA practical step-by-step guide to set up an AI analytics Slack bot with an open source framework so your team can chat with data directly in Slack.
Guides· Mar 2026nao
How to Set Up an AI Analytics Teams Bot with an Open Source FrameworkA step-by-step guide to set up an AI analytics Microsoft Teams bot with an open source framework so your team can chat with data directly in Teams.
Guides· Mar 2026nao
4 Steps to Improve Your Analytics Agent Reliability from 45% to 86%A practical case study on how context engineering, dbt documentation, and data-model fixes improved an analytics agent from 45% to 86% reliability.
Guides· Mar 2026Pinterest Engineering Blog
Unified Context-Intent Embeddings for Scalable Text-to-SQLPinterest describes evolving from basic text-to-SQL to an analytics agent using context-intent embeddings that capture the analytical intent behind historical queries.
Guides· Mar 2026nao
How to Build Your In-House Analytics Agent Fully with Open SourceA practical 7-step framework to build your in-house analytics agent fully with open source tooling, from context engineering to evaluation and rollout.
Guides· Mar 2026nao
5 Steps to Deploy an Analytics Agent on dbt MCP to Your Whole CompanyA practical 5-step setup guide to deploy an analytics agent on dbt MCP with nao, from choosing the right MCP to rolling out chat with data across your company.
Guides· Mar 2026nao
How to Build an AI-First Data TeamA practical 5-step guide to turning your team into an AI-first data team with context engineering, open source analytics, and chat with data workflows.
Guides· Feb 2026nao
How Data Teams Build Skills for Agentic Analytics: 7 Practical StepsA practical 7-step guide for data teams to design, organize, and test skills that improve analytics agent reliability and chat-with-data outcomes.
Guides· Feb 2026nao
How to Evaluate an Analytics Agent: A Practical Guide with nao testA step-by-step guide to evaluating your analytics agent's reliability using nao's built-in unit test framework, from writing your first test to running the visual dashboard.
Research· Feb 2026nao
What Is the Impact of a Semantic Layer on Analytics Agent Performance?We tested MetricFlow semantic layers against a plain rules.md file to measure the real impact on analytics agent reliability, cost, and speed. Here is what the data says.
Research· Feb 2026nao
What Context Has the Most Impact on Analytics Agent Performance?We tested schema, data sampling, profiling, dbt repos, and rules.md to find which context pieces actually improve analytics agent reliability. Here is what the data says.
Opinion· Feb 2026Learn Analytics Engineering
The Truth of Self-Service AnalyticsArgues self-service analytics needs cultural change and well-modeled data, not just tooling, and that transparent AI agents only help when the underlying data models are solid.
Research· Feb 2026nao
5 Best Open Source Analytics Agents (2026 Comparison)Compared 5 open source analytics agents on SQL accuracy, context depth, and production readiness. Find the right stack for your data team.
Opinion· Feb 2026nao
Why data teams need an open framework for context engineeringAnalytics agents won't scale without a proper context stack. Here is why context engineering needs its own open framework, and what that framework looks like.
Opinion· Feb 2026nao
5 Reasons Natural Language Analytics is Replacing Traditional BI DashboardsWhy natural language interfaces are displacing traditional BI dashboards, and how to prepare your data warehouse for this shift.
Guides· Feb 2026nao
How to Build Production-Ready AI Agents for Data Analytics: The Complete GuideA complete guide to building reliable AI agents for data analytics, from context engineering to production deployment.
Research· Feb 2026nao
20 Best AI Analytics Agents Compared: 2026 BenchmarkA full benchmark of 20 AI analytics agents on accuracy, cost, context depth, and data team UX. Warehouse-native tools, AI-native BI, and open-source agents compared.
Opinion· Feb 2026Metabase Blog
Lessons learned from building AI analytics agents: build for chaosMetabase's team recounts how their Metabot agent failed in a demo and argues that reliable analytics agents come from context engineering across the agent lifecycle, not prompt engineering, and must be built for the chaos of real production data.
Guides· Jan 2026Hamel's Blog
LLM Evals: Everything You Need to KnowA comprehensive FAQ on LLM evals covering error analysis, evaluator design, annotation, tooling, and production deployment.
Opinion· Jan 2026Preset Blog
The Semantic Layer Is Back. Here's What We're Doing About It.Argues semantic layers are regaining relevance because AI agents need structured governance, and lays out Preset's approach of supporting multiple semantic layer platforms.
Guides· Oct 2025nao
Basics of Git data analysts should learnLearn Git essentials for data professionals, including practical workflows and how AI can help automate version control for SQL, Python, and data projects.
Guides· Oct 2025nao
How to choose the right data stackA comprehensive guide to choosing the right data stack for your company's stage and data maturity.
Guides· Sep 2025Anthropic Engineering
Effective context engineering for AI agentsAnthropic's practical guide to curating the finite context window so agents stay reliable, covering tool design, retrieval, and compaction.
Guides· Sep 2025nao
How to Set Up dbt Core (2026): Your First Model with AILearn to set up dbt Core in 2026. Install the CLI, write staging and fact models on BigQuery, add tests, and use AI to build 10x faster.
Opinion· Aug 2025benn.substack.com
The Context LayerStancil examines how the semantic layer and AI agents might converge into a shared context layer that holds an organization's business knowledge.
Guides· Aug 2025pedronasc.com
Lessons on building an AI data analystA practitioner's field notes arguing that a real AI data analyst needs multi-step planning and validated code generation rather than plain text-to-SQL, with a semantic layer to encode business logic.
Research· Aug 2025Data Engineering Weekly
Knowledge, Metrics, and AI: Rethinking the Semantic LayerA wide-ranging conversation defining the semantic layer as a knowledge graph plus a compiler, and arguing it is the missing infrastructure for trustworthy AI analytics.
Research· Aug 2025Lightdash
Why we're building an open semantic layer at LightdashThe case for a portable, open semantic layer so that metric definitions stay consistent across every BI tool and AI agent that consumes them.
Opinion· Aug 2025benn.substack.com
Can analysis ever be automated?Argues that unlike code, analytical outputs can't be verified by real-world testing, creating a catch-22 for AI analysts where trust rests on reputation rather than provable correctness.
Guides· Jul 2025Airbyte Blog
Semantic Layer: What It Is and Why It Matters?A guide to how semantic layers unify data sources and centralize metric definitions, including how they ground generative AI with context to reduce hallucinations.
Research· Jul 2025Chroma Research
Context Rot: How Increasing Input Tokens Impacts LLM PerformanceAn empirical study of 18 frontier models showing performance degrades non-uniformly as input length grows, challenging the assumption that bigger context windows are always better.
Guides· Jul 2025LangChain Blog
Context Engineering for AgentsOrganizes context engineering for agents into four strategies (write, select, compress, isolate) that keep long-running agents reliable.
Research· Jun 2025Databricks Blog
AI/BI Genie is now Generally AvailableDatabricks announces general availability of AI/BI Genie, its conversational analytics agent, covering the semantic knowledge store, deep research capability and preview adoption.
Guides· May 2025Google Cloud Blog
Getting AI to write good SQL: Text-to-SQL techniques explainedA Google Cloud engineer explains techniques for context building, table retrieval, LLM-as-a-judge evaluation, and prompting to improve text-to-SQL accuracy.
Opinion· May 2025From an Engineer Sight
The Implication of Malloy in AnalyticsArgues that Malloy, as a semantic modeling and query language, lets LLMs handle complex analytical questions better than raw text-to-SQL by encoding business logic and consistent meaning.
Opinion· May 2025benn.substack.com
No, really, everything becomes BIStancil argues that AI is shifting from presenting information to making decisions, and that data tools are racing to feed agents organizational context.
Research· May 2025Cube Blog
Why Agentic AI Needs a Semantic CoreArgues that a semantic layer is essential infrastructure for AI agents doing analytics, translating business questions into accurate governed queries and preventing hallucinations from raw text-to-SQL.
Research· Apr 2025dbt Developer Blog
Introducing the dbt MCP Server: Bringing Structured Data to AI Workflows and AgentsIntroduces dbt's open-source MCP server as the glue letting AI agents discover, query, and execute against the dbt Semantic Layer and project metadata.
Guides· Jan 2025huyenchip.com
AgentsA long practical primer on how LLM agents work, breaking down tools, planning, and the failure modes that make them hard to ship.
Guides· Dec 2024Anthropic
Building Effective AgentsAnthropic's engineering team argues the most reliable agents come from simple, composable patterns rather than heavy frameworks.
Research· Nov 2024arXiv
Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL WorkflowsAn enterprise-grade benchmark of real-world text-to-SQL workflow problems over BigQuery, Snowflake, and Postgres databases with thousands of columns and multiple dialects.
Guides· Oct 2024Hamel's Blog
Using LLM-as-a-Judge For Evaluation: A Complete GuideA step-by-step guide, drawn from 30+ AI implementations, to building and validating LLM judges that align with human judgment.
Guides· Sep 2024Uber Engineering Blog
QueryGPT - Natural Language to SQL Using Generative AIUber details QueryGPT, its LLM plus vector-search system with domain workspaces that generates SQL from natural language and cut query authoring time significantly.
Research· Aug 2024Snowflake Blog
Cortex Analyst: Paving the Way to Self-Service Analytics with AISnowflake introduces Cortex Analyst, a text-to-answer service built on a semantic model that lets business users query data in natural language, detailing how the semantic layer boosts accuracy.
Guides· May 2024O'Reilly Radar
What We Learned from a Year of Building with LLMs (Part I)Six practitioners share tactical lessons on prompting, RAG, evals, and guardrails from a year of shipping real LLM applications.
Research· May 2024arXiv
CHESS: Contextual Harnessing for Efficient SQL SynthesisProposes a multi-agent pipeline that retrieves relevant schema and context to synthesize SQL efficiently over large, complex databases.
Guides· Apr 2024Pinterest Engineering Blog
How we built Text-to-SQL at PinterestPinterest engineers describe using RAG-based table retrieval to build a production text-to-SQL feature inside their Querybook analytics tool.
Guides· Mar 2024Hamel's Blog
Your AI Product Needs EvalsArgues most failing LLM products share a root cause (no robust evaluation system) and lays out a framework of unit tests, human/model evaluation, and A/B testing.
Guides· Mar 2024eugeneyan.com
Task-Specific LLM Evals that Do & Don't WorkCovers which evaluation metrics actually work for classification, summarization, and other tasks, including factual-consistency evals as hallucination guardrails.
Guides· Feb 2024AWS Machine Learning Blog
Build a robust text-to-SQL solution generating complex queries, self-correcting, and querying diverse data sourcesAWS walks through a text-to-SQL architecture combining RAG over database metadata, a multi-step self-correction loop, and Athena as the execution engine.
Research· Feb 2024arXiv
Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User QueriesStudies how the choice of physical database schema and data model affects text-to-SQL system accuracy using real user queries.
Research· Dec 2023davidsj.substack.com
We told you soEarly evidence that pairing an LLM with a semantic layer beats raw text-to-SQL by a wide margin on real questions.
Guides· Jul 2023eugeneyan.com
Patterns for Building LLM-based Systems & ProductsDetails seven practical patterns for reliable LLM products: evals, RAG, fine-tuning, caching, guardrails, defensive UX, and user feedback.
Research· May 2023arXiv
Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLsIntroduces BIRD, a large-scale cross-domain text-to-SQL benchmark of 12,751 question-SQL pairs over 95 real databases, showing a wide gap between LLM and human execution accuracy.
Research· Apr 2023arXiv
DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-CorrectionShows that decomposing text-to-SQL into sub-tasks with in-context learning and self-correction substantially improves LLM accuracy on the Spider benchmark.
Guides· Apr 2023huyenchip.com
Building LLM Applications for ProductionAn early, widely-cited overview of the hard parts of productionizing LLM apps: prompt reliability, cost, latency, and task composition with agents and tools.
Opinion· Feb 2023MotherDuck
Big Data is DeadThe former BigQuery engineering lead argues most workloads are small, so big-compute distributed systems are no longer needed for the vast majority of teams.
Opinion· Apr 2021benn.substack.com
The missing piece of the modern data stackThe foundational essay arguing the modern data stack lacks a centralized metrics layer to make metric definitions consistent across every downstream tool.