RAG Tutorials

The problem RAG solves

An LLM only "knows" what was in its training data, up to a fixed cutoff date, and it has no built-in access to your private documents, tickets, or internal wikis. Two options exist to bridge that gap: fine-tuning the model on your data (expensive, slow to update, and still no guarantee of factual recall), or Retrieval-Augmented Generation (RAG) — retrieving the relevant information at query time and handing it to the model as context, so it generates its answer grounded in that material instead of guessing from memory alone.

The RAG pipeline, step by step

• 1. Ingestion — collect the source documents you want the system to be able to answer from (docs, tickets, PDFs, wikis, code).
• 2. Chunking — split documents into smaller passages sized to fit well within the embedding model and the LLM's context window, since retrieving whole documents is usually too coarse and too large.
• 3. Embedding — convert each chunk into a numeric vector that captures its semantic meaning, using an embedding model.
• 4. Indexing / storage — store those vectors in a vector database optimized for fast similarity search (see Vector Database in the glossary).
• 5. Retrieval — at query time, embed the user's question the same way, and find the chunks whose vectors are most similar (commonly via cosine similarity).
• 6. Augmentation — insert the retrieved chunks into the prompt as context, alongside the user's original question.
• 7. Generation — the LLM produces an answer grounded in the retrieved context, ideally citing which passage supports which claim.

Types of RAG

• Naive RAG — a single retrieval pass followed by generation. Simple, fast, and a reasonable starting point, but can retrieve irrelevant or incomplete context for complex questions.
• Advanced RAG — adds refinements such as query rewriting (reformulating the user's question for better retrieval), re-ranking (a second pass to reorder retrieved chunks by relevance), and hybrid search (combining keyword search with vector similarity).
• Agentic RAG — an agent (see AI Agent Tutorials) decides when to retrieve, what to search for, and whether to retrieve again or from a different source, rather than following one fixed retrieval step. Useful for multi-hop questions that need information gathered iteratively.

Evaluating a RAG system

• Retrieval relevance — did the system actually retrieve the passages that contain the answer?
• Faithfulness / groundedness — does the generated answer accurately reflect the retrieved context, without adding unsupported claims?
• Answer correctness — is the final answer actually right, measured against a known-correct reference where possible.

A RAG system can fail at any one of these independently — good retrieval with poor generation still produces a wrong answer, and vice versa, so it's worth evaluating retrieval and generation quality separately when debugging.

Best practices

• Tune chunk size deliberately — too large and irrelevant content dilutes the context; too small and you lose necessary surrounding information.
• Use hybrid search — combining keyword and vector search catches cases either approach misses alone (e.g. exact error codes or IDs that embeddings handle poorly).
• Filter by metadata — date, source, document type, or permissions can narrow retrieval before similarity search even runs.
• Require citations — have the model reference which retrieved passage backs each claim, and treat unverifiable claims with suspicion.
• Enforce access control at retrieval time — never rely on the model itself to decide what a user is allowed to see (see AI Security).
• Keep the index fresh — a stale index quietly reintroduces the exact "outdated knowledge" problem RAG was meant to solve.

Common pitfalls

• Chunking without regard to document structure, splitting a table or code block mid-way through.
• No re-ranking step, so the top-similarity chunk isn't actually the most useful one for the question.
• Treating RAG as a complete hallucination fix — a model can still misinterpret or misstate retrieved context, so faithfulness still needs to be checked.
• Ignoring retrieval latency at scale — large indexes need proper vector search infrastructure, not a linear scan.