A general-purpose language model has read a vast amount of public text, but it hasn't read your HR policies, your product manuals, your contracts, or last quarter's incident reports. Ask it about those and it will do what these models do when they don't know: produce something fluent and plausible that may be wrong.
Retrieval-augmented generation — RAG — is the most common way to close that gap. If you're a leader being asked to approve an "AI knowledge assistant" project, there's a good chance it's RAG underneath. This guide explains it without the jargon, so you can ask sharper questions.
The idea, with an analogy
Imagine two exam formats. In a closed-book exam, a student answers from memory, and when memory fails, they bluff. In an open-book exam, the student looks up the relevant pages first, then writes the answer using them.
RAG turns the language model's exam into an open-book one. Before the model answers, a search step finds the most relevant passages from your own documents and places them in front of the model along with the question. The model then writes its answer based on that supplied text, ideally with references back to where it came from.
How it works, in three steps
1. Prepare the knowledge. Your documents are split into manageable chunks, and each chunk is converted into a numerical representation called an embedding, which captures meaning so that similar ideas end up close together. These are stored in a searchable index, often a vector database. We explain the underlying idea in embeddings: teaching machines what words mean.
2. Retrieve at question time. When someone asks a question, the system converts it the same way and finds the chunks whose meaning is closest. Good systems often combine this semantic search with traditional keyword search and then re-rank the results.
3. Generate with the context. The retrieved passages are added to the prompt, and the model composes an answer grounded in them, with instructions such as "answer only from the provided text; if it isn't there, say so."
For the engineering side, see building a RAG pipeline with OpenAI and building RAG applications with LangChain.
Why companies choose RAG
- Grounded answers. The model draws on your actual documents, which reduces — though never entirely eliminates — invented answers.
- Fresh knowledge. Update the documents, and answers update, without retraining a model.
- Traceability. Answers can cite their sources, so users can verify them.
- Access control is possible. Retrieval can respect who's allowed to see what, if it's designed to.
- Cost and speed. It's usually quicker and cheaper to build than training or heavily customizing a model.
RAG, fine-tuning, or just a long prompt?
| Approach | Best for | Watch out for |
|---|---|---|
| RAG | Answering from large, changing document sets, with citations | Retrieval quality is the ceiling on answer quality |
| Fine-tuning | Teaching a consistent style, format, or specialized behavior | Doesn't reliably add new facts; needs training data and upkeep |
| Long context | Small document sets that fit in one prompt | Cost and latency grow with every request |
These aren't mutually exclusive, and many production systems combine them. Our post on fine-tuning vs prompting explains the trade-offs in more depth.
Where RAG projects go wrong
Most failures are not about the AI model. They're about the data and the plumbing:
- Poor source documents. Outdated, duplicated, contradictory, or badly scanned files produce poor answers. RAG surfaces the state of your knowledge; it doesn't repair it.
- Bad retrieval. If the right passage isn't retrieved, the model can't use it. Chunking strategy, metadata, and search tuning matter more than model choice.
- Permission leaks. If the index ignores document permissions, an assistant can reveal content to people who shouldn't see it. This is the most serious risk in enterprise deployments.
- No evaluation. Teams ship after trying a few questions, then are surprised by real-world failures. You need a test set of realistic questions with known good answers, run on every change.
- Unclear scope. An assistant that's expected to answer "anything" about the company will disappoint. A well-defined domain — say, IT support or HR policy — works far better.
- No feedback loop. Without a way for users to flag bad answers and for owners to fix source content, quality decays.
Questions leaders should ask
- What exactly will it answer from, and who owns those sources? Someone must be accountable for keeping them current.
- How does it respect existing access permissions?
- How will we measure answer quality, and what's the threshold for launch?
- What happens when it doesn't know? The right behavior is to say so, not to guess.
- Can answers be traced to sources? Citations build trust and enable audit.
- Where does our data go, and is it used to train anyone's models?
- Who reviews it when it's wrong, and how fast can we fix it?
A good first project
Pick a narrow, high-volume, document-heavy question set — internal IT help, policy questions, product support for a single product line. Define 50 to 100 realistic test questions with correct answers up front. Build, measure against that set, and expand only when the numbers justify it. This is the same incremental logic we describe in a practical roadmap for AI adoption.
Build shared understanding
RAG projects succeed when business owners, data teams, and engineers share a vocabulary and a definition of "good." Our Retrieval-Augmented Generation (RAG) in Practice course gives technical teams hands-on experience, while Generative AI at Work and AI for Everyone help the wider organization use these tools effectively. If you're scoping a project and want a second opinion, get in touch.