This is the fourth post in our OpenAI API fundamentals series, building on context and cost management.
Ask a model a question about your company's internal policy document, your product's actual API reference, or anything else it never saw during training, and it will still often produce a confident, plausible-sounding, and completely fabricated answer — because generating plausible text is exactly what it's designed to do, whether or not it actually has the relevant facts. Retrieval-Augmented Generation fixes this by finding the actually-relevant text first, and explicitly handing it to the model as context before asking it to answer.
The core idea in one sentence
Instead of asking the model "what's our refund policy?" and hoping it somehow knows, RAG retrieves the actual relevant passage from your real documentation first, and asks the model to answer using that specific passage — turning an open-book question the model might guess at into a closed-book one it answers directly from provided material.
Step 1: embeddings, and what they actually represent
response = client.embeddings.create(
model="text-embedding-3-small",
input="Our refund policy allows returns within 30 days of purchase.",
)
vector = response.data[0].embedding
print(len(vector))1536
An embedding is a list of numbers (a vector) representing a piece of text's meaning in a way that supports mathematical comparison. Two passages discussing similar ideas produce vectors that are numerically close together; two passages about unrelated topics produce vectors that are numerically far apart — even if they don't share a single word in common. This numerical closeness is the entire mechanism RAG's retrieval step relies on.
Step 2: indexing your actual documents
documents = [
"Our refund policy allows returns within 30 days of purchase.",
"Corporate training sessions can be rescheduled up to 48 hours in advance.",
"Cloud architecture courses require basic networking knowledge as a prerequisite.",
]
doc_embeddings = [
client.embeddings.create(model="text-embedding-3-small", input=doc).data[0].embedding
for doc in documents
]For a handful of documents, comparing embeddings directly in Python is genuinely fine. For anything beyond that — thousands or millions of document chunks — a vector database (Pinecone, Weaviate, or pgvector as a Postgres extension are common choices) stores embeddings and performs this similarity search efficiently at real scale, which plain in-memory comparison stops handling well fairly quickly.
Step 3: retrieval — finding what's actually relevant
import numpy as np
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
def retrieve(query, documents, doc_embeddings, top_k=2):
query_embedding = client.embeddings.create(
model="text-embedding-3-small", input=query
).data[0].embedding
scores = [cosine_similarity(query_embedding, doc_emb) for doc_emb in doc_embeddings]
ranked = sorted(zip(scores, documents), reverse=True)
return [doc for _, doc in ranked[:top_k]]
relevant_docs = retrieve("Can I get my money back?", documents, doc_embeddings)
print(relevant_docs)["Our refund policy allows returns within 30 days of purchase.", ...]
Notice the query — "Can I get my money back?" — shares almost no words with the retrieved document, "Our refund policy allows returns within 30 days of purchase." This is the actual payoff of embeddings over naive keyword search: the retrieval is matching on meaning, not shared vocabulary, which is what makes RAG work for real user questions phrased however a real person happens to phrase them.
Step 4: generation — answering, grounded in what was retrieved
def answer_with_rag(query, documents, doc_embeddings):
relevant = retrieve(query, documents, doc_embeddings)
context = "\n".join(relevant)
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": f"Answer the user's question using only the following context. "
f"If the answer isn't in the context, say you don't know.\n\n{context}",
},
{"role": "user", "content": query},
],
)
return response.choices[0].message.content
print(answer_with_rag("Can I get my money back?", documents, doc_embeddings))Yes — our refund policy allows returns within 30 days of purchase.
"If the answer isn't in the context, say you don't know" in the system prompt is doing genuinely important work, not just filling space — without it, the model will often still fabricate a plausible-sounding answer even when the retrieved context doesn't actually contain one, defeating much of the point of building a retrieval step in the first place.
Chunking: a decision that matters more than it first appears to
Real documents are usually far longer than a single embedding call should ideally handle at once, which means splitting them into smaller chunks before embedding each one separately. Chunk size is a genuine trade-off, not an arbitrary implementation detail: chunks that are too large dilute a specific relevant sentence with a lot of surrounding, less-relevant text, hurting retrieval precision; chunks that are too small lose the surrounding context a passage needs to be properly understood on its own. Most real RAG systems land somewhere in the range of a few hundred tokens per chunk, often with a small overlap between consecutive chunks so a fact split across a chunk boundary isn't lost entirely in either piece.
What to actually remember from this post
- RAG retrieves relevant text first, then asks the model to answer using it — turning an open-book guess into a closed-book, grounded answer.
- Embeddings represent meaning, not vocabulary — retrieval works even when a query shares no words with the relevant document.
- A vector database becomes necessary at real scale; in-memory comparison is genuinely fine for a small, fixed document set.
- Explicitly instruct the model to say "I don't know" when context doesn't contain an answer — without it, fabrication persists even with retrieval in place.
Next in the series: Fine-Tuning vs. Prompting: When Each Approach Makes Sense, the final post in this series — covering the other major lever for adapting a model's behavior to your specific use case.