Enterprise RAG in Production: Why Pilots Stall and What Fixes Them

A RAG demo is built to impress: a few hundred clean documents, questions the team already knows the answers to, and a reviewer who reads the first paragraph. Production reverses every condition. The corpus spans every system the company runs, most of it under permissions; documents change and get deleted; and the people asking know when an answer is wrong. Retrieval-augmented generation was first described in 2020 as a way to give a language model an external memory it could cite and update,¹ and it lives or dies on the retrieval half of its name.

• Nearly 1 in 3respondents in McKinsey's 2025 survey reported consequences from AI inaccuracy, the most reported AI risk²
• 17 to 33%of answers from commercial RAG legal research tools were hallucinated in a 2024 Stanford-led study³
• 60%of AI projects unsupported by AI-ready data that Gartner expects organizations to abandon through 2026⁴

Why enterprise RAG pilots stall in production

Enterprise RAG pilots stall when the system meets the conditions the demo left out: exact terms, permissions, change, deletion and questions the documents cannot answer. In McKinsey's 2025 survey of 1,993 participants, inaccuracy was the AI risk respondents most often reported consequences from, and one of two risks that most respondents say their organizations are working to mitigate.² In a July 2024 Gartner survey of 1,203 data management leaders, 63% of organizations lacked, or were unsure they had, the right data management practices for AI; Gartner lists vector stores, chunking and embedding among the practices to add.⁴

The clearest public evidence comes from law, where answers can be checked. Researchers at Stanford and Yale ran a preregistered set of more than 200 legal questions through three commercial RAG research tools from providers that had promoted RAG as avoiding or eliminating hallucinations. The tools hallucinated on 17 to 33% of answers, against 43% for a general-purpose chatbot with no retrieval.³ When the authors traced causes, poor retrieval contributed to 20 to 47% of the hallucinated answers depending on the tool, and citing an inapplicable source, such as an overruled case or the wrong jurisdiction, to 23 to 38%.³ Those are index, ranking and metadata problems. In the systems we build and review, the same pattern recurs: the model faithfully summarizes the wrong passage.

The seven production failure modes

A RAG pilot fails in production in seven recurring ways, and each has a symptom users report, a cause in the pipeline and a fix an engineer can schedule.

Six of the seven sit in retrieval and data. A larger model fixes none of them, which is why a stalled pilot needs a pipeline review before it needs a new model.

Fix retrieval first: hybrid search, reranking and chunking by structure

Retrieval quality sets the ceiling on answer quality, so fix it before touching prompts or models: combine keyword and vector search, rerank the merged candidates, and give the model a few passages cut along the document's own structure. The BEIR benchmark tested ten retrieval systems across 18 datasets outside their training domain and found the classic keyword method, BM25, a strong baseline, dense vector models often behind other approaches, and reranking models best on average, at a higher compute cost.⁵ Enterprise text, full of product codes, clause numbers, acronyms and names, is out of domain for any off-the-shelf embedding model; keyword search matches those terms exactly, and vectors blur them.

Run both searches, merge the two ranked lists with reciprocal rank fusion, and let a cross-encoder reranker order the top few dozen candidates. Then pass the model fewer passages. More context feels safer and performs worse: a 2023 study found that language models often use information best when it sits at the beginning or end of the input, and significantly worse when it sits in the middle of a long context, even for models built for long inputs.⁶ Keep the handful of passages that clear a relevance threshold, and put the strongest first.

Chunk by structure. Fixed-size windows cut a clause from its exception and a table from its header row. Split on the document's own headings, sections, list items and table rows. Attach the title, section path, effective date, owner and source link to every chunk, and keep a pointer to the parent section so the generator can see the surrounding rule when a small chunk matches. Scanned PDFs, slide decks and spreadsheets each need their own parser, and in the systems we build, parsing decides more of the final quality than the embedding model does.

Permission-aware retrieval: filter by the user's entitlements at query time

Permission-aware retrieval means every search runs as the person asking, and only passages that person could open in the source system are eligible to reach the model. OWASP's 2025 Top 10 for LLM applications lists vector and embedding weaknesses as a risk in its own right, naming unauthorized access through misaligned access controls and leaks between users who share a vector database, and it recommends permission-aware vector stores.⁷ The common mistake: a pilot indexes a document library with a service account that can read everything, and the assistant becomes a search box over every file that account could open. Filtering the answer afterwards is too late, because the restricted text is already in the model's context.

1. 1Identify the user Resolve the signed-in user's groups and roles, including nested groups, from the OAuth or OpenID Connect token your identity provider issues.
2. 2Filter inside the search Apply the entitlement filter within both the keyword and the vector query, so restricted chunks never become candidates. Filtering after retrieval returns thin results and wastes the top k.
3. 3Re-check the most sensitive sources For HR, legal and board material, confirm access against the source system at query time, because the access list copied into the index can be hours old.
4. 4Log the retrieval Record which passages were retrieved for whom, under which policy version, so an access question can be answered from the record.

Never index a source whose access controls you cannot read; if a system cannot export them, leave it out or index it only for a group allowed to see all of it. Sync permission changes on a stated interval and treat that interval as a security commitment, because a revoked user keeps access until the next sync. Key any answer cache by entitlement set, or one user's cached answer becomes another user's leak. In a multi-tenant system, keep each client in its own index or namespace.

Freshness, versioning and deletion

A production index must reflect each source within a stated time, know which version of a document is current, and forget what the source deletes. Nightly full reindexing is a demo habit. Drive ingestion from change events or modification timestamps in each source, hash every chunk so unchanged text is not embedded again, and publish a freshness target per source: minutes for the ticketing system, a day for the policy library. Give every chunk an effective date and a superseded-by link, and retrieve the current version by default. The inapplicable-authority errors in the Stanford study, such as citing an overruled case, are the legal form of this failure.³

Deletion has to be designed in from the start. Under Article 17 of the GDPR, a person can require a controller to erase their personal data without undue delay where one of the listed grounds applies, and Article 12 requires the controller to report the action taken within one month of the request, extendable by two further months for complex or numerous requests.⁸ In a RAG system that data lives in far more places than the source.

Ask your vector store's supplier two questions: when a deleted vector stops being returned, and when it is physically removed. Write both answers into the retention policy. Fine-tuning is harder: a fact learned into model weights has no row to delete, and removing it reliably means retraining.

Retrieve the text or call the system

Retrieve text when the answer lives in prose written to be read; call the system of record when the answer is a live value, a calculation over records, or depends on row-level permissions. Order status, account balances, inventory, entitlements and case history change by the minute and already sit behind an API with its own authorization model. Exporting them to text and embedding the result creates a stale copy with the permissions stripped out. Give the model a tool instead: a typed call to the system's API, executed with the user's authority, returning structured data the answer can cite.

The Model Context Protocol is an open protocol that gives models a standard way to reach those tools and data sources; its current specification is dated July 28, 2026.¹⁰ RAG and MCP work together: a retrieval service can itself be offered to the model as an MCP tool, next to tools that query ServiceNow, Salesforce or SAP. Our guide to MCP and A2A in the enterprise covers how to govern those connections. When the model decides for itself when to search and which tool to call, the pattern is called agentic RAG.

RAG vs fine-tuning vs long context

Use RAG for knowledge that changes, must be cited or depends on who is asking; use fine-tuning to change how a model behaves; use long context for one-off work over a small, known set of documents.

A 2023 comparison of knowledge injection methods found that RAG consistently outperformed unsupervised fine-tuning, both for knowledge the model had seen in training and for entirely new facts, and that models struggle to learn new facts through unsupervised fine-tuning.¹¹ Long context has improved without removing the position problem. On the NoLiMa benchmark, published in 2025, 11 of 13 models that claim at least 128K tokens of context fell below half their short-context score at 32K tokens, and even one of the strongest dropped from 99.3% to 69.7%.¹² Long context is a good way to read one contract. It is a poor way to search ten thousand.

Evaluate retrieval separately from generation

Score retrieval and generation separately, because an end-to-end score cannot tell you which half to fix. Build the test set from real questions in the pilot's logs, each labeled with the passages that answer it, and add two kinds the demo never had: questions the corpus cannot answer, and questions a given test user is not permitted to see answered.

Low recall means the answer never reached the model, and no prompt will fix it. High recall with low faithfulness means the model had the right passage and drifted from it. Our guide to LLM and AI agent evaluation covers building the graded set and wiring it into your release pipeline.

Citations and "I don't know"

Every claim in a production answer should cite the passage it came from, and the system should say it does not know when no passage supports an answer. Enforce both in code. Ask the model to cite passage ids, then check that every cited id was in the retrieved set and that any quoted text appears in that passage; strip or flag a claim that fails. Link each citation to the source section, at the right version. Base the abstain rule on evidence: when no passage clears the reranker's threshold, the assistant says what it searched, says it found no answer, and routes the question to the owner of that content.

Tune abstention against both errors. In the Stanford study, a tool that tied for the lowest hallucination rate answered accurately on only 20% of queries, because 63% of its answers were refusals or lacked grounding.³ A system that rarely invents and rarely helps will stall as surely as one that invents. Track both rates, and send every unanswered question to the content owners as a gap report.

What drives the cost per answer

The cost of an answer is driven by the tokens sent to the model, the number of model and retrieval calls each answer takes, and the ingestion work spread across all answers. Model it before scale with a formula like this one:

cost per call   = input tokens x input price + output tokens x output price
                  (input = instructions + passages + conversation history)

cost per answer = query embedding + keyword and vector search
                + candidates reranked x rerank price
                + calls per answer x cost per call        (1 for single-shot RAG, more for agentic)
                + monthly ingestion and index cost / monthly answers
                + review minutes per answer x reviewer cost

Four levers move it. Passing five strong passages instead of twenty cuts input tokens and keeps the relevant one out of the middle of a long context. Caching answers to repeated questions cuts calls, provided the cache is keyed by entitlement set. Routing simple lookups to a smaller model lowers the price per call. Agentic retrieval multiplies calls per answer, so reserve it for the questions that need several steps.

What to ask before an enterprise RAG system goes live

Ask for a measured answer to each of these seven questions before go-live; any question without one marks work still to do.

This is the core of our AI development work: when we take over a stalled pilot, the first step is a retrieval and permission review against a graded set, so the fix goes where the failure is.