Text-to-SQL in 2026: How Accurate AI Data Analysts Are, Where They Fail and How to Deploy Them

Every data team has a queue of questions from people who cannot write SQL. Text-to-SQL, the use of a language model to turn a plain-English question into a database query, promises to empty that queue, and every major data platform now sells a version of it as an AI analyst or data agent. The pitch rests on accuracy figures that are hard to compare, and the risks sit where a demo does not look: business definitions that live in people's heads, database credentials, and a warehouse that bills by the byte. This guide covers what independent benchmarks show, what vendors and named companies report, where the answers go wrong, what the security record and the rules say, what it costs and how to build an AI data analyst that holds up.

• 10.8%of questions from real data warehouse logs answered correctly by leading agent frameworks using GPT-5.2¹
• 52.8%of problems in a widely used text-to-SQL benchmark subset had annotation errors in their answer keys⁴
• 71%of data professionals name incorrect or hallucinated outputs reaching stakeholders as a top concern¹⁴

What independent benchmarks show

The benchmarks that look most like company data tell a consistent story. BEAVER, built by researchers at MIT and elsewhere from real data warehouse query logs, has 9,128 question and query pairs over 812 tables in 19 domains. In its 2026 version, agent frameworks using GPT-5.2 answered 10.8% correctly, and 30.1% when given hints for every subtask.¹ On LiveSQLBench, a set refreshed to stay out of training data, the best plain models score 35% to 37% and the best agent system 48%.² When questions are ambiguous and the system has to ask follow-ups, results fall further: on BIRD-Interact, GPT-5 completed 8.67% of tasks in conversation and 17% when it could act as an agent.¹⁵

The leaderboards quoted in marketing look very different, and the gap is the lesson. Spider 2.0, built from enterprise workflows on BigQuery and Snowflake with databases that often hold more than 1,000 columns, launched with o1-preview solving 21.3% of tasks.¹⁷ Its Snowflake board is now topped by an engineered agent at 96.70, while the benchmark's own reference agent running Claude 4 Sonnet scores 25.78.³ On BIRD, the best test score is 82.95% against 92.96% for human data engineers and students.¹⁶ Schema exploration, retrieval of examples, voting across candidate queries and documentation turn a 25 into a 90. The model is one part of the product.

Why the top scores overstate accuracy

The top scores also measure agreement with answer keys that are often wrong. Researchers at the University of Illinois found annotation errors in 52.8% of BIRD Mini-Dev problems and 62.8% of the Spider 2.0-Snow problems they checked. Re-scoring 16 open-source agents on corrected labels moved their ranks by up to nine places, and the corrected ranking correlated only weakly with the official one.⁴ Execution accuracy, the standard metric, compares result sets, so it passes some wrong queries and fails some right ones; an expert-style model judge raised agreement with human reviewers from a Cohen's kappa of 62 to 87.04.¹⁸ A 2026 claim of human-level accuracy on BIRD applies to an expert-verified re-annotation of the set, not to the official test.¹⁹ A leaderboard rank shows which system best reproduces a benchmark's labels, which is a different question from how often your finance team will get the right number.

What vendors claim

The vendors agree on one thing: accuracy depends on curated context. Each platform asks for a semantic model or metric definitions, verified example queries and written instructions before it performs well. Google's BigQuery documentation calls the underlying models an early-stage technology that "can generate output that seems plausible but is factually incorrect."²⁵

What companies that built their own report

Named deployments are more modest, and all are self-reported. Salesforce says its internal Horizon Agent "only had the correct response ~50% of the time" at launch.²⁶ Uber's QueryGPT cut the time to write a query from about 10 minutes to about 3; in a limited release it averaged about 300 daily users, 78% of whom said it saved them time.²⁷ LinkedIn's SQL Bot is often described as 95% accurate, but that figure is a survey: about 95% of users rated its accuracy "Passes" or above and about 40% rated it "Very Good" or "Excellent." Its most used feature was "Fix with AI" for debugging queries, at 80% of sessions.²⁸ Swiggy raised its Hermes assistant's SQL accuracy from 54% to 93% by storing past queries with their prompts and retrieving similar ones.²⁹

Demand is the clearest result. Ramp's in-house agent answered 1,476 questions in four weeks, against 66 answered in its data help channel over the same period.³⁰ Grab avoided free-form generation for its reports: its analysts were mostly running the same queries with different dates and filters, so it turned those queries into tools the model can call.³¹ For recurring reporting, a library of approved queries with parameters is the safer design.

Context matters more than the model

The independent evidence on why curation works is narrow but consistent. An April 2026 paired study ran 100 questions on a retail dataset through Claude Opus 4.7, Claude Sonnet 4.6 and GPT-5.4. A 4 KB hand-written document of measures, conventions and disambiguation rules lifted accuracy by 17 to 23 points, to 67.7% to 68.7%, and the three models were statistically indistinguishable with or without it. In the authors' words, "model choice within tier does not" explain the difference.⁵ An often-quoted 2023 study by researchers at data.world, a knowledge graph vendor, found GPT-4 at 16% on enterprise SQL and 54% over a knowledge graph of the same database.³²

dbt Labs' benchmark shows the trade-off. Through its Semantic Layer, questions inside the model scored 98.2% to 100%, and questions outside it scored 0% until the model was extended, because the layer declined to answer them. In dbt's words, "With text-to-SQL, failure looks like a plausible but incorrect answer. With the Semantic Layer, failure looks like an error message."²³ Few teams have done this work. Of the 30% of data professionals in dbt's 2025 survey who use AI to answer questions in natural language, two-thirds use plain SQL generation and one-third a semantic layer.³³

Where text-to-SQL goes wrong

The errors research measures are mostly about meaning. In BEAVER's first error analysis, models given the right tables failed because they did not know database-specific rules or custom functions (41.5% of failures), linked the question to the wrong table or column (39.0%), or added or dropped columns (22.0%).³⁴ Real schemas are far larger than benchmark ones. Microsoft researchers describe an internal finance warehouse with 632 tables and more than 4,000 columns, and a product usage dataset with 2,281 tables and 65,000 columns. They found 41.1% of questions in one public benchmark ambiguous, and GPT-3.5 recognized a question the database could not answer only 60% of the time.³⁵ On ambiguous questions, the best model tested on the AMBROSIA benchmark found a valid reading 31% of the time, against 66% on clear ones.³⁶

The dangerous failure is a query that runs and returns a believable wrong number. In an August 2026 study on an insurance schema, direct SQL generation produced 29 wrong runs that executed without error, and seven more that matched the expected result only by coincidence in the data, out of 114. The authors note that SQL "can execute successfully while using the wrong relationship role or aggregation grain."⁶ People are a weak check. In a study of 26 participants given queries that contained errors, users fixed about 56% of them on average, and explanations, visualizations and a chat assistant made no significant difference.⁷ That study predates current models and its participants knew they were looking for errors, so a business user reading a confident chart is unlikely to do better. In dbt's 2026 survey, 71% of data professionals named incorrect or hallucinated outputs reaching stakeholders as a top concern.¹⁴

The security record

Research shows the pattern is general. Prompt-to-SQL injection worked across seven models in applications built on LangChain, and a 2026 study found "substantial security vulnerabilities" in four open-source data agents and two cloud analytics services.⁴¹⁴² OWASP's guidance is to limit what an agent can call "to only the minimum necessary" and to execute actions "in the context of that specific user."⁴³ Any table that holds free text, such as tickets, CRM notes or reviews, is a channel for injected instructions.

Managed products handle identity better than most self-built agents, with one trap to check. Databricks says Genie evaluates data access "using each end user's own Unity Catalog permissions."⁴⁴ Snowflake semantic views use owner's rights, so "a user with access to a semantic view does not require separate access to its underlying tables"; masking and row access policies on the tables still apply, but the grant on the view becomes the real gate.⁴⁵ A self-built agent that connects through one service account loses every per-user policy unless it passes the user's identity to the database.

What it costs

Seat prices cap the AI fee but not the warehouse bill underneath. A December 2025 study ran 180 generated queries from six models on 230 GB of BigQuery data. Non-reasoning models showed cost variance of up to 3.4 times, with outliers above 36 GB per query against the best model's 1.8 GB average, mostly from missing partition filters and inefficient joins.¹² The usual safeguards have gaps. BigQuery lets you cap the bytes billed per query, and notes that on tables without clustering a LIMIT clause does not reduce cost.⁵² Google's conversational analytics API dry-runs each query and stops it "without incurring a charge" when it would scan more than the cap.⁵³ Snowflake's resource monitors "work for warehouses only" and cannot track AI services, and its statement timeout defaults to two days.¹³⁵⁴ An agent warehouse left on defaults has no practical ceiling.

Rules that apply

How to build an AI data analyst that holds up

Start narrow. Uber organizes QueryGPT into workspaces of curated tables and sample queries for each business domain,²⁷ and Databricks advises aiming for five or fewer tables and an agent that answers "questions for a particular topic and audience."⁶¹ Write the business definitions down in a semantic layer such as dbt MetricFlow, LookML, Snowflake semantic views or Databricks metric views, and keep them in version control in a form you can move, since the vendor-neutral exchange format, now Apache Ossie, is still at its v0.1 specification.⁶² Add example queries signed off by the people who own the numbers; Databricks recommends SQL expressions for business semantics and example SQL queries alongside text instructions.⁶¹

Retrieve before generating. LinkedIn pulls the top 20 candidate tables and has a model re-rank them to 7 before writing the query,²⁸ and the CHESS research system cut tokens fivefold while gaining about 2% accuracy by pruning large schemas.⁶³ Classify each question before answering. Snowflake rejects ambiguous questions "upfront, rather than responding with potentially misleading answers."⁶⁴ Show the SQL, the metric definition and any assumption with every answer, and let the system say it cannot answer.

The controls follow from the incidents. Connect through a database role with no write grants, never an application-level read-only flag. Run each query as the person asking so row and column policies apply, and mask personal columns. Dry-run every query against a byte cap, set a timeout in minutes, and put a separate budget on AI services. Run model-written chart or Python code in a sandbox or not at all, and review the code of any MCP server before it touches production.

Test on your own questions. LinkedIn, Uber and Snowflake each built sets of real questions with expert-approved answers. LinkedIn accepts several correct queries per question because single answers "underreported recall by 10-15%," and its model judge scores within one point of a human 75% of the time.²⁸ Genie supports up to 500 benchmark questions per agent, and Snowflake's evaluations flag verified queries that used to pass and now fail.⁶⁵⁶⁶ Databricks expects benchmarks above 80% before user acceptance testing.⁶⁷

1. 1Collect real questions Pull six months of requests from the data team's queue and query logs, and group them by domain and how often they recur.
2. 2Pick one domain Start with a domain of a few tables, clear owners and moderate stakes, such as operations or support reporting, and leave finance and regulated reporting for last.
3. 3Write the definitions down Model the measures, joins and business rules in a semantic layer, and add example queries signed off by the owners of each number.
4. 4Lock down access and cost Connect through a read-only database role that runs as the user, mask personal columns, cap bytes per query and set short timeouts.
5. 5Build the test set Turn 100 to 150 real questions into a benchmark with expert answers, allow more than one correct query, and set a pass rate to clear before users see it.
6. 6Release to analysts first Put the assistant inside the tools analysts already use, show the SQL and definitions with each answer, and collect their corrections.
7. 7Widen access as the numbers allow Add business users and new domains only while the test set and the false-success rate on live questions stay above your bar after every change.

Text-to-SQL now works well inside a narrow domain whose meaning has been written down, and poorly on a raw warehouse, whatever the model. The scarce input is the time of the people who own the numbers to define metrics and approve example queries. Companies that treat that work as the project, with access and cost controls and a test set drawn from their own questions, get an analyst people trust. Those that point a chat window at the warehouse get confident numbers nobody can check.

This is how we approach AI analytics in our data engineering work: start with one domain and the people who own its numbers, write the definitions into a semantic layer you keep, connect the agent through read-only roles that run as each user, and test every release on your own questions before anyone relies on the answers.