Retrieval-Augmented Generation Explained for Builders

Every RAG system has two halves: indexing, which runs ahead of time, and querying, which runs on each question. Anthropic summarizes the classic version: break the corpus into chunks "usually no more than a few hundred tokens," convert each chunk into a vector embedding that encodes meaning, store the vectors in a database that supports similarity search, then at query time find the closest chunks and add them to the prompt (Anthropic).

Indexing (offline)
  documents -> clean and parse -> chunk -> add context -> embed -> vector index
                                                       -> keyword (BM25) index
Querying (per request)
  question -> embed + keyword search -> merge results -> rerank -> top chunks
           -> prompt with chunks and citations -> model -> answer with sources

Each arrow is a decision with a quality cost if you get it wrong. Parsing loses tables and headings. Chunking splits an answer across two pieces. Embedding misses exact identifiers. Retrieval returns near misses. The model then writes a confident answer from whatever it was given. The rest of this article walks through the decisions that matter most.

A chunk is the unit you retrieve. If the answer to a question is split across two chunks, or buried in a chunk full of unrelated text, retrieval struggles no matter how good the model is.

Pinecone's guide frames the choice around four questions: what kind of content you are chunking, which embedding model you use, how long and complex the expected queries are, and how the results will be used (Pinecone, Chunking strategies). Short content such as product descriptions may not need chunking at all. Long documents with headings and sections should usually be split along that structure.

The common strategies:

Pinecone recommends starting with fixed-size chunking and iterating "only after determining it insufficient." NVIDIA's chunking experiments explain why iteration matters. Across several datasets, the best strategy differed even between similar financial document types. Very small and very large chunks generally underperformed medium sizes, and factoid questions favored smaller chunks while analytical questions favored larger ones or whole pages (NVIDIA, Finding the best chunking strategy). In NVIDIA's tests, page-level chunking gave the most consistent results across datasets, but the broader lesson is that no single setting wins everywhere.

Practical defaults that hold up:

• Split on the document's own structure first: headings, pages, sections.
• Keep tables, lists and code blocks whole.
• Store metadata with each chunk: source document, title, section, date and access level.
• Test two or three chunk sizes against your own questions before settling.

Chunking creates a quiet problem. A passage that made sense in its document loses meaning on its own. Anthropic's example is a chunk from a company filing that reads "The company's revenue grew by 3% over the previous quarter." Out of context, it does not say which company or which quarter, so a question about that company's growth may never retrieve it (Anthropic).

Anthropic's fix, which it calls contextual retrieval, prepends a short explanation to each chunk before embedding and indexing it. A model reads the whole document and the chunk, then writes one or two sentences situating the chunk, such as which filing, company and period it comes from. The contextualized chunk is then indexed for both vector and keyword search.

Original chunk:
  The company's revenue grew by 3% over the previous quarter.

Contextualized chunk:
  This chunk is from ACME Corp's quarterly filing for Q2; the previous
  quarter's revenue is stated earlier in the filing.
  The company's revenue grew by 3% over the previous quarter.

In Anthropic's own tests, as published in that write-up and read on 2026-10-17, contextual embeddings combined with contextual keyword search reduced the rate of failed retrievals by 49 percent, and adding a reranking step raised the reduction to 67 percent. The extra context costs one model call per chunk at indexing time, not at query time, and caching the full document across calls keeps that cost down.

If you cannot run a model over every chunk, cheaper versions of the same idea still help: prepend the document title and section heading to each chunk, and store dates and product names as metadata you can filter on.

Vector search finds chunks with similar meaning. It is good at conceptual questions and poor at exact terms. A Towards Data Science guide on production RAG gives the classic failure: an engineer searches for "dead-letter queue threshold configuration," but the dense vector for the right paragraph has averaged that exact phrase together with everything else in it (Towards Data Science, Hybrid search and re-ranking). Anthropic's version is an error code: a vector search for "Error code TS-999" may return general content about error codes and miss the one page that mentions TS-999.

Keyword search, usually the BM25 ranking function, has the opposite profile. It rewards rare exact terms and cannot match synonyms or paraphrases. Hybrid search runs both and merges the results, usually with rank fusion or a weighted blend. The Towards Data Science guide tuned the blend on a labeled set of helpdesk questions and found that both pure modes did worse than a mix. It also stresses that the right weighting depends on your corpus and should be measured, not guessed.

Reranking adds a second, more careful pass. The first stage uses fast retrieval to pull a wide set of candidates. The second stage uses a cross-encoder, a model that reads the question and each candidate together, to score them precisely. The Towards Data Science guide explains the trade: a cross-encoder is much more accurate than comparing two independent vectors, but it must run once per candidate at query time, so you cannot apply it to the whole corpus. "Use the bi-encoder to cast a wide net... then use the cross-encoder to re-score only those N candidates precisely."

def retrieve(question, user):
    dense = vector_index.search(embed(question), k=50, filter={"acl": user.groups})
    sparse = bm25_index.search(question, k=50, filter={"acl": user.groups})
    candidates = reciprocal_rank_fusion(dense, sparse)[:100]
    ranked = reranker.score(question, candidates)
    return ranked[:10]

Anthropic's summary of its own experiments is a useful checklist: embeddings plus keyword search beat embeddings alone, adding context to chunks helps a lot, reranking beats no reranking and "all these benefits stack" (Anthropic). How many chunks to pass to the model is also worth testing. More chunks raise the chance the answer is included but can distract the model.

Once you have the right passages, the prompt should make them easy to use correctly:

• Wrap each chunk with its source. Include a document title and identifier so the model can cite it.
• Tell the model to answer only from the provided passages and to say when they do not contain the answer.
• Ask for citations that point to specific sources, so users can verify the answer, which IBM identifies as a main driver of trust in RAG systems (IBM).
• Keep instructions separate from retrieved text. Retrieved documents are data, and a document that contains instructions should not change the model's behavior.

Permissions must be enforced at retrieval time. AWS notes that RAG lets developers "restrict sensitive information retrieval to different authorization levels," and its guidance includes filtering results to the documents the end user is allowed to see (AWS). Store an access label with every chunk and filter on it in the query itself, as in the code above. Never retrieve everything and ask the model to hold back what the user should not see.

Freshness needs a plan too. When a source document changes, re-chunk and re-embed it, and remove the old chunks. A RAG system that answers from last year's policy is worse than one that admits it does not know.

When a RAG answer is wrong, there are two possible causes: the right passage was not retrieved, or it was retrieved and the model misused it. You can only tell them apart if you measure each stage.

For retrieval, build a labeled set of real questions, each paired with the document or chunk that answers it. Then track:

The Towards Data Science guide used exactly this approach, a labeled set of questions drawn from helpdesk history scored on hit rate and MRR, to tune its hybrid search weighting (Towards Data Science). Anthropic's experiments measured the share of relevant chunks missing from the top 20 results (Anthropic).

For answers, check whether each claim is supported by the retrieved passages, whether the answer covers the key facts and whether the citations point to the right sources. Run both sets of checks every time you change the chunking, the embedding model, the retrieval settings or the prompt.

If you use Quanta Intelligence agents, the same discipline applies to any knowledge you connect: structure it well, keep it current and test the questions your users actually ask.