What Is RAG? A Beginner-Friendly Guide to How It Works and What It Does

"I want ChatGPT to consult our company manual and answer questions." "I want it to search and summarize new research papers." For these tasks, a model's learned knowledge alone may not be enough. You can provide the material it needs when it answers.

One way to do this is RAG (Retrieval-Augmented Generation). It retrieves relevant information from external material and adds it to the context the model uses to answer. Ready-made AI features can also consult documents, but uploading a file does not always trigger the same retrieval process.

This article breaks down RAG in three diagrammed steps and walks through vector databases, a LangChain implementation, and when to choose RAG over fine-tuning --- at a level a beginner can follow, but technically accurate enough to be useful.

1. What Is RAG --- Retrieval-Augmented Generation

RAG (Retrieval-Augmented Generation) literally means "generation that has been augmented by retrieval."

In one sentence, RAG is a system in which an LLM (large language model) uses relevant information retrieved from external sources when generating an answer. The diagrams here show the basic pattern: retrieve first, then answer. Retrieval can use existing databases or keyword search; a vector database is not mandatory.

A Cooking Analogy

An LLM on its own is "a chef who cooks from memory." Talented, sure, but they can't make a dish they don't know, and they have no idea what's in your fridge.

RAG is the system that "hands the chef a cookbook and tells them what's in the fridge before they start cooking." Now the chef can flip through the recipes and use the ingredients on hand to put together the right dish.

What "Retrieval," "Augmented," and "Generation" Each Do

The key idea: instead of retraining the LLM itself, you hand it the "knowledge it needs" from the outside, every time a question comes in. That's the fundamental difference between RAG and fine-tuning, which we'll cover later.

LangChain's retrieval guide explains this basic pattern and how existing SQL databases or document systems can serve as sources.

2. Why You Need RAG --- 3 Limits of LLMs Alone

There are three problems that ChatGPT, Claude, and other LLMs can't solve on their own.

Limit 1: The Knowledge Cutoff (How Fresh the Information Is)

A model's learned knowledge alone cannot verify information that has changed since training. Knowledge cutoffs differ by model. Later events need to be checked against documents or search results.

• "Tell me about the new product announced yesterday." -> Check the announcement
• "What changed in the terms of service last week?" -> Check the new terms
• "What's today's exchange rate?" -> Check timestamped data

RAG can use information retrieved from news, databases, or APIs. But outdated sources can still lead to outdated answers. Documents must be kept current, and retrieval results need checking.

Limit 2: Hallucinations (Plausible-Sounding Lies)

When you ask an LLM about something it doesn't know, it has a strong tendency to make up an answer that looks reasonable. This is called hallucination.

For example, if you ask about a company's leave allowance without supplying its private policy, the model may invent a plausible number. This is an illustration, not a response executed or observed for this article.

RAG retrieves actual documents to help ground answers in evidence. If you retain source metadata, you can show document names or page numbers alongside the answer. A displayed citation does not prove that the answer is correct: retrieval may miss relevant material, or the model may misread it.

Limit 3: No Access to Internal or Private Data

Do not assume a public model knows your private company manuals or contracts. Before providing documents, check your permission to use them and where they will be sent.

RAG selects relevant material to give the model; it does not guarantee confidentiality by itself. Even with your own database, external embedding or LLM APIs receive documents, questions, or retrieved passages. Check user permissions at retrieval time, destinations, storage, and logs. The FAQ explains these distinctions.

3. How It Works --- RAG in 3 Steps

RAG operates in two broad phases: "preparation (indexing)" and "runtime (Q&A)."

Preparation Phase --- Vectorize and Store Your Documents

1. Collect documents: gather the PDFs, Word files, HTML, Markdown, or whatever else you want to use
2. Chunk them: split the documents into reasonable lengths (say, 500-1000 characters)
3. Embed them: run each chunk through an embedding model (e.g., OpenAI text-embedding-3-small) to convert it into a vector --- an array of numbers, often 1536-dimensional
4. Store in a vector DB: save the chunks alongside their vectors in a dedicated database (Pinecone, Qdrant, etc.)

You run this whenever new documents are added or existing ones get updated.

Runtime Phase --- 3 Steps to Answer a Question

Here's what happens when a user asks something:

1. Step 1: Retrieval
   • Vectorize the question using the same embedding model
   • Retrieve the top K relevant chunks from the vector DB (3 in this example)
   • Closeness is typically measured with cosine similarity
2. Step 2: Augmented
   • Embed the retrieved chunks into the prompt as "reference information"
   • Something like "Use the following information to answer the question: [search results] Question: [user's question]"
3. Step 3: Generation
   • The LLM (GPT, Claude, Gemini, etc.) generates the answer with the reference material in hand
   • Add citations for "which document this came from" as needed

Concrete Example: Asking ChatGPT About Your Company's Leave Policy

The leave allowances, service periods, and article numbers below belong to a fictional company policy used to explain the mechanism. They do not describe an actual entitlement.

The flow for a question like "How many days of paid leave do I get?":

1. The question gets vectorized by the embedding model -> [0.12, -0.45, 0.78, ...]
2. The vector DB returns 3 chunks related to "leave" and "paid time off"
3. Retrieved chunks: "Article 15: Annual Paid Leave. Employees with 6 months of service receive 10 days...", "Up to 20 days based on tenure...", etc.
4. Prompt assembly: "Reference: Article 15... Question: How many days of paid leave do I get?"
5. The LLM answers: "10 days at 6 months of service, up to 20 days based on tenure (see Employment Rules, Article 15)"

4. The Key Components of a RAG System

Let's walk through the five pieces that make up a RAG.

1. Embedding Model

An AI model that turns text into a numerical vector. It's trained so that "semantically similar texts end up close together in vector space."

See the official embedding guide for OpenAI's default dimensions and how to shorten them. Compare candidates using retrieval results on your documents, rather than dimensions alone.

2. Vector Database

A database or extension that stores vectors and retrieves similar ones. The next section explains it in more detail.

3. Retriever

Often combines vector search with keyword search (BM25 and friends) or hybrid search.

4. LLM (the Generator)

The large language model that produces the final answer --- GPT, Claude, Gemini, Llama, and so on. Works with both commercial APIs and self-hosted OSS models. Every one of these families keeps replacing its versions, so check the current model names in our list of major AI models.

5. Prompt Template

The template that combines search results and the user's question into a single message for the LLM. A surprisingly important piece for RAG accuracy.

You are an assistant who knows our internal policies inside out.
Answer the question using only the reference information below.
If the answer isn't in the reference, reply with "I don't have that information."

[Reference]
{retrieved_chunks}

[Question]
{user_question}

[Answer]

5. What Is a Vector Database?

A vector database provides indexes and features for storing high-dimensional vectors and retrieving similar ones. You can use a dedicated service or extend an existing database, such as PostgreSQL with pgvector. Speed and accuracy depend on the index, data volume, and settings.

Major Vector DBs Compared

Which One Should You Pick?

The Pinecone and Weaviate prices above were checked against official pricing pages on October 9, 2026. Free software still requires hosting and maintenance resources. Cloud costs depend on capacity, retrieval volume, extra AI features, backups, and other options.

• Try retrieval in Python: Chroma or a similar tool (prepare the packages and models)
• Already running PostgreSQL: pgvector (keep the database in one place)
• Let a service manage infrastructure: a managed service such as Pinecone (indexes, capacity, and permissions still need configuration)
• Self-host: tools such as Qdrant or Weaviate (manage operations and permissions yourself)
• Large document collections: evaluate candidates such as Milvus with real data for index behavior, capacity, and retrieval performance

For more on picking where to host all this, take a look at PaaS (Vercel, etc.) vs Shared Hosting, VPS, and Cloud.

6. Possible Uses of RAG

RAG can support systems that answer using internal documents, product information, and other sources. Here are examples of possible uses.

Use Case 1: Internal Document Q&A (Knowledge Base)

Retrieve employment policies, operating manuals, technical specifications, meeting notes, or sales materials to inform answers to employees. The design must restrict documents by user permissions and reflect document changes in the index.

Use Case 2: Customer Support Automation

Build RAG over FAQs and support history, then automate first-touch responses with a chatbot. Human operators get to focus on the complex tickets.

Use Case 3: Specialist Q&A in Law and Medicine

Retrieve case law, medical papers, or guidelines and summarize information for specialists to review. Show sources and have people verify the material's applicability and the answer. Citations alone do not guarantee a correct or safe decision.

Use Case 4: Research Paper Search and Summarization

Retrieve papers or abstracts you are permitted to use to inform questions such as "What are the trends in this research area?" or "Which studies relate to this method?" State the scope and publication dates of the retrieved material, then check results and research conditions against the originals.

Use Case 5: E-Commerce Product Search and FAQ

RAG that integrates product manuals, reviews, and return policies. Now you can do natural-language search like "is this vacuum any good with pet hair?"

Use Case 6: Developer Documentation Chat

Retrieve a library's official documentation to answer questions such as "Can you show sample code for this operation?" Include the relevant version in retrieval conditions so that incompatible documentation is not mixed.

Use Case 7: Internal Codebase Search and Explanation

Retrieve code and design documents to inform questions such as "How do I use this function?" or "Which files implement similar logic?" Referencing code is also useful in GitHub Copilot Chat and developer AIs such as Cursor and Claude Code, but their internal processing is not necessarily identical to this diagram.

Use Case 8: Document Guides Such as llms.txt

llms.txt is a proposal for providing a text guide to a site's material. It can help with finding and collecting documents, but it does not guarantee that AI systems will read them or cite them correctly.

7. RAG vs Fine-Tuning --- Which Should You Pick?

The other approach to "give an LLM custom knowledge" that comes up alongside RAG is fine-tuning. The two take fundamentally different paths.

The Core Difference

When RAG Is the Right Pick

• Knowledge changes frequently (news, internal docs, product info)
• You need to show evidence for the answer (law, medicine, finance)
• Different questions need different documents, and you want to evaluate retrieving relevant passages rather than sending everything
• You want to prototype document-based answers without retraining the model

When Fine-Tuning Is the Right Pick

• You need a specific style/tone (brand voice, character)
• You want the model to learn specialty-domain language patterns (medical, legal)
• You want to evaluate savings from shorter prompts or a smaller model, including training costs
• You can prepare examples of desired responses and a separate evaluation dataset

Fine-tuning availability depends on the service. OpenAI's official documentation, checked on October 9, 2026, says fine-tuning is not available to new users. Check supported models and contract terms first.

Combine and Evaluate When Needed

RAG and fine-tuning can be combined. For example, additional training can adjust response format while retrieval supplies updated documents. Combining them does not guarantee better quality.

Start by defining expected answers and an evaluation method, then test whether better instructions or added documents are sufficient. Consider RAG when reference material is needed, and fine-tuning when you need to adjust response behavior. Compare cost, speed, and quality on actual inputs. OpenAI's optimization guidance also begins with evaluation and prompt improvements, with fine-tuning for appropriate tasks. Whether a fine-tuned model can run locally depends on weight distribution, licensing, and the execution environment.

8. Before You Build Your Own --- Test with Your Own Documents

Before building RAG yourself, check with your own documents whether "search, then answer" is useful at all. It saves detours, and in most cases the first check needs no code.

Step 1: Check whether you can supply the whole document collection

If all your material fits within the chosen model's context window, you can also supply it without retrieval. In its 2024 Contextual Retrieval explanation, Anthropic suggested putting collections below 200,000 tokens (about 500 pages) directly in the prompt and using prompt caching for efficiency (official explanation). That 200,000-token figure is not a threshold shared by all models. Decide using the current model's input limit, document length, answer quality, and cost.

Step 2: Put the documents into a ready-made "Projects" feature and ask questions

Claude's Projects switch to RAG automatically when project knowledge approaches the context limit, handling up to 10x more content. It is a paid-plan feature; when it turns on, an indicator appears and Claude uses a "project knowledge search" tool when answering (Claude Help Center). ChatGPT's Projects also let you add files and ask about them, with limits such as the number of files set per plan.

Step 3: Test with 10 questions you already know the answers to

Write 10 questions whose answers you know the location of in the documents, and record three things.

1. Is the answer correct?
2. Is the passage it cites as evidence the right one?
3. When you ask about something that is not in the documents, does it say so?

If the third check produces invented answers, repeat the test after instructing the model to say when information is absent. Ten questions are an initial trial, not proof of production suitability or safety, even if most answers are correct. Extend evaluation to important questions, similar documents, contradictions, and access permissions.

Step 4: Compare custom and managed options when more control is needed

As one example of retrieval improvements, Anthropic reported that Contextual Embeddings combined with Contextual BM25 reduced top-20-chunk retrieval failure rates by 49%, or 67% with reranking (its experiment conditions and results). These are retrieval measurements on the vendor's evaluation data, not a guarantee that errors in your answers will fall by the same percentage.

The minimal build-it-yourself version is in the next section; production improvements such as hybrid search and reranking are covered in the RAG and vector database implementation guide.

9. How to Build It --- A RAG with LangChain

Here are the major frameworks, followed by a minimal Python example.

Major Frameworks

Minimal RAG with LangChain

This learning example retrieves passages from a PDF and supplies them to a model to answer. LangChain 1.x requires Python 3.10 or later. Set an API key in the OPENAI_API_KEY environment variable and prepare access to the OpenAI API models used. Do not put the key in code or a public repository.

This example sends PDF text and questions to the embedding API, then retrieved passages and questions to the LLM API. Running Chroma locally does not make all processing local. Check API billing and usage terms separately from your ChatGPT subscription, and use nonconfidential documents you are permitted to send. This code is a learning example whose operation has not been verified. Before running it, check it against the official usage for the library version you use.

# pip install -U langchain-openai langchain-chroma langchain-text-splitters langchain-community pypdf
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_chroma import Chroma

# 1. Load the document
loader = PyPDFLoader("rules.pdf")
docs = loader.load()

# 2. Split into chunks
splitter = RecursiveCharacterTextSplitter(
    chunk_size=500, chunk_overlap=50
)
chunks = splitter.split_documents(docs)

# 3. Embed + build the vector DB
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Chroma.from_documents(chunks, embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

# 4. Retrieve (take the 3 chunks closest to the question)
question = "How many days of paid leave do I get?"
found = retriever.invoke(question)
context = "\n\n".join(d.page_content for d in found)

# 5. Pass the evidence and generate the answer
llm = ChatOpenAI(model="gpt-6-luna")
answer = llm.invoke([
    {"role": "system", "content": "Answer using only the material below. If the answer isn't in it, say it isn't in the material.\n\n" + context},
    {"role": "user", "content": question},
])
print(answer.content)
print("Sources:", [d.metadata for d in found])

The example is designed to retrieve relevant PDF passages and supply them to GPT-6 Luna to answer. It also prints metadata from the retrieved documents. Page information depends on the source and loader; citations in the generated answer need separate verification.

gpt-6-luna is the ID checked on the official model page on October 9, 2026. Before using it, check supported APIs, terms, and pricing in the official model list. Compare models on the same documents and questions for evidence interpretation, answer quality, speed, and cost.

The code uses the LangChain 1.x package structure. In LangChain 1.0, older examples using langchain.chains, including RetrievalQA, moved to the separate langchain-classic package. This example imports text splitting from langchain_text_splitters and implements the official "2-Step RAG" pattern: retrieve first, then pass the evidence to the model. Check import paths and Python requirements in the official migration guide. Record package versions and verify the implementation in a reproducible environment.

What a More Production-Ready Build Adds

• Better chunking (semantic splits, hierarchical chunks, etc.)
• Hybrid search (vector + BM25 keyword search)
• Reranking (Cohere Rerank, voyage-rerank, etc., to reorder search results)
• Query rewriting (HyDE, Multi-Query, and similar techniques to improve recall)
• Evaluation pipeline (automated evaluation with RAGAS)

10. RAG Challenges and How to Handle Them

In production, check the following aspects of RAG.

Challenge 1: Chunking Is Hard

How you split your documents has a big impact on retrieval accuracy. Too short and you lose context; too long and search precision drops.

What to do:

• Semantic chunking (split on meaning boundaries)
• Use overlap (let adjacent chunks share some text)
• Hierarchical chunks (store as parent/child; search children, return parents)

Challenge 2: Retrieval Accuracy

You'll grab chunks that look relevant but aren't, and miss critical information.

What to do:

• Hybrid search (vector + BM25 keyword)
• Reranking models to reorder after retrieval
• Multi-query generation (search the same question phrased multiple ways)

Challenge 3: Context Length Limits

There's a cap on the number of tokens you can hand the LLM. You can't shove in unlimited chunks.

What to do:

• Tighten K (top 3-5 chunks)
• Summarize first, then pass
• Use a long-context LLM (the limit differs by model, and some now take around a million tokens; check the limit for your model in the official docs)

Challenge 4: Evaluation Is Hard

Measuring RAG answer quality objectively isn't easy. Building a ground-truth dataset is its own problem.

What to do:

• Use RAGAS (an OSS evaluation framework for RAG)
• Automate metrics like answer correctness, answer relevance, retrieval faithfulness
• LLM-as-a-Judge (have a different LLM grade the output)

Challenge 5: Multilingual and Multimodal Content

Documents that mix English and Spanish, PDFs with images, tables and charts --- all tricky.

What to do:

• Use multilingual embedding models (BGE-M3, Cohere Multilingual)
• Pre-extract text from images and tables with an LLM (OCR + VLM)
• Multimodal embeddings (CLIP, Nomic, etc.)

11. Major Tools and Services

Here are the main tools you'd reach for, grouped by category.

Frameworks and Libraries

• LangChain --- integrations for retrieval, models, and data sources
• LlamaIndex --- focused on data connections
• Haystack --- enterprise-oriented
• DSPy --- automatic prompt optimization

Vector DBs (Managed)

• Pinecone --- managed vector search
• Weaviate Cloud --- GraphQL support
• Qdrant Cloud --- the managed Qdrant service
• Zilliz Cloud --- managed Milvus

Vector DBs (OSS / Self-Hosted)

• Chroma --- vector retrieval accessible from Python
• Qdrant --- vector retrieval and filtering
• Weaviate --- OSS edition
• Milvus --- a vector database
• pgvector --- PostgreSQL extension

Embedding Models

• OpenAI text-embedding-3 --- generate embeddings through an API
• Voyage AI --- introduced in Anthropic's docs
• Cohere Embed --- multilingual
• BGE-M3 --- a multilingual candidate for local use

No-Code and Managed RAG Services

• ChatGPT Projects / existing Custom GPTs --- ready-made features for consulting documents; check each feature's processing and usage terms
• Claude Projects --- Anthropic's built-in RAG
• Notion AI --- search across documents inside Notion
• Microsoft Copilot (Microsoft 365) --- a candidate for consulting internal material; check contract, connections, and permissions
• Dify --- OSS no-code AI building platform
• Vertex AI Agent Builder --- Google Cloud's RAG-building service
• Amazon Bedrock Knowledge Bases --- AWS managed RAG

Evaluation Tools

• RAGAS --- OSS RAG evaluation framework
• TruLens --- general LLM app evaluation
• LangSmith --- official LangChain tracing and evaluation

For more on fine-tuning, see what is fine-tuning too — it explains when to use it versus RAG and methods like LoRA/QLoRA, for beginners.

FAQ

Pricing, product availability, and API data-processing descriptions were reviewed against official sources on October 9, 2026. Code execution, API calls, charges, and confidentiality in a real environment were not tested. Check each service's latest terms before implementing.