What actually makes a RAG system better

Retrieval-augmented generation is one of those phrases that has aged faster than the technique it describes. Two years ago, RAG meant chunk a document, embed it, stuff the nearest neighbors into the prompt, and hope. Today, the bar is higher — and most of the systems calling themselves "RAG" are still doing the 2023 version of it.

I've spent the last year building Recall, an internal RAG system for the kind of work I do for clients. What I've learned, in the most compressed form I can manage, is this: the gains aren't where people are looking. Embeddings improved, vector databases got cheaper, models got bigger — and the real wins came from somewhere else entirely.

Here's what actually moves the needle.

1. Better chunking, not bigger embeddings

The first thing every team optimizes is the embedding model. It's also the thing with the smallest payoff. A well-chunked corpus retrieved with text-embedding-3-small will outperform a poorly-chunked one with the most expensive model on the market.

The naive version of chunking looks like this — and it's almost always wrong:

# Don't do this.
def chunk_naive(text, size=500):
    return [text[i:i+size] for i in range(0, len(text), size)]

It splits mid-sentence, mid-word, sometimes mid-acronym. The retriever then tries to find meaning in fragments that don't carry any. What you want instead is semantic chunking with overlap, respecting sentence boundaries, and carrying metadata forward:

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=120,
    separators=["\n\n", "\n", ". ", " ", ""],  # try paragraph first, fall back gracefully
    length_function=len,
)

chunks = splitter.create_documents(
    texts=[doc.text],
    metadatas=[{
        "source": doc.id,
        "section": doc.section,        # so the retriever can filter
        "updated_at": doc.updated_at,  # so stale content can be deprioritized
    }],
)

Most RAG systems retrieve worse than they should because they're searching a haystack of arbitrary 500-token slices. Fix the slices, and everything downstream gets better for free.

2. Hybrid retrieval beats vector-only

Pure vector search is good at semantic matching and bad at exact matching. If your user types a product SKU, an error code, or a person's name, a vector store will happily return semantically-related-but-wrong results. The fix is not exotic: combine BM25 (or any keyword retriever) with the vector search, and merge.

from rank_bm25 import BM25Okapi
import numpy as np

def hybrid_search(query, chunks, vector_store, alpha=0.5, k=50):
    # Dense (semantic) scores
    dense_hits = vector_store.similarity_search_with_score(query, k=k)
    dense_scores = {c.id: s for c, s in dense_hits}

    # Sparse (keyword) scores
    bm25 = BM25Okapi([c.text.split() for c in chunks])
    sparse_scores = dict(zip(
        [c.id for c in chunks],
        bm25.get_scores(query.split())
    ))

    # Normalize and blend
    def norm(d):
        vals = np.array(list(d.values()))
        return {k: (v - vals.min()) / (vals.ptp() + 1e-9) for k, v in d.items()}

    dense_n, sparse_n = norm(dense_scores), norm(sparse_scores)
    combined = {
        cid: alpha * dense_n.get(cid, 0) + (1 - alpha) * sparse_n.get(cid, 0)
        for cid in set(dense_n) | set(sparse_n)
    }
    return sorted(combined.items(), key=lambda x: -x[1])[:k]

The alpha parameter is the lever — 0.5 is a fine starting point, but tune it on your own eval set. Almost every production system that punches above its weight is doing some version of this.

3. Rerankers are the cheapest 20% you'll buy

Take the top 50 results from your retriever. Run them through a cross-encoder reranker. Keep the top 5. That's the entire intervention, and it routinely beats a year of fiddling with embedding models:

import cohere

co = cohere.Client(api_key)

def rerank(query, candidates, top_n=5):
    results = co.rerank(
        query=query,
        documents=[c.text for c in candidates],
        top_n=top_n,
        model="rerank-english-v3.0",
    )
    return [candidates[r.index] for r in results.results]

# In the pipeline:
candidates = hybrid_search(query, chunks, vector_store, k=50)
top_chunks = rerank(query, candidates, top_n=5)

Rerankers are slow per-document, which is why nobody uses them as the primary retriever — but they're cheap on a small candidate set, and they read the interaction between the query and the document rather than their separate embeddings. That's a different operation, and it's the one that actually matches what a human means.

4. Query rewriting before retrieval

Users don't write queries the way documents read. They write "why is my invoice wrong" and the answer is buried in a doc titled "Billing reconciliation error codes." No embedding model on earth will save you from that gap.

The fix is a small LLM call before retrieval:

from anthropic import Anthropic
client = Anthropic()

def rewrite_query(user_query, n=3):
    resp = client.messages.create(
        model="claude-haiku-4-5",
        max_tokens=300,
        messages=[{
            "role": "user",
            "content": f"""Rewrite this question into {n} alternative phrasings
that a relevant document might use. One per line, no numbering.

Question: {user_query}"""
        }],
    )
    return [user_query] + resp.content[0].text.strip().split("\n")

# Retrieve for each rewrite, union the results
queries = rewrite_query(user_query)
all_hits = []
for q in queries:
    all_hits.extend(hybrid_search(q, chunks, vector_store, k=20))

# Dedupe and rerank against the *original* query
unique = {h[0]: h for h in all_hits}.values()
top_chunks = rerank(user_query, list(unique), top_n=5)

It's an extra 200ms and it doubles recall on the queries that matter.

5. Stop pretending the LLM will catch your mistakes

The biggest mistake in RAG design is assuming the generation step will gracefully handle bad retrieval. It won't. If you hand the model irrelevant context, it will either hallucinate confidently or — worse — produce an answer that blends the irrelevant context with its priors in ways that are very hard to debug.

Treat retrieval as the load-bearing step. The LLM is the thing on top, not the thing covering for you.

What this implies for AI search

The reason this matters for SEO is that every AI Overview, every ChatGPT citation, every Perplexity answer is running its own RAG pipeline against the web — and the same five problems show up at internet scale. Brands that win generative search aren't winning because the models like them; they're winning because their content survives chunking, gets retrieved on hybrid queries, ranks well under a reranker, matches rewritten queries, and gives the model something solid enough that it doesn't have to improvise.

In other words: the discipline of making a RAG system better and the discipline of making a brand findable inside an LLM are turning out to be the same discipline. Which is probably why the people building one are quietly becoming pretty good at the other.