Paul Simon offered fifty ways to leave your lover; I can offer ten to feed a language model, which is fewer but considerably more actionable. "RAG" gets talked about as if it were a single technique — embed your documents, look up the nearest neighbors, paste them into the prompt. That naive version is real, and it's where everyone starts, but it's one point in a design space. The interesting decisions are about what you retrieve, how you decide relevance, and how many times you go back to the well before answering. Here are ten distinct architectures, what each mechanism actually does, when it earns its keep, and what it costs.
1. Naive top-k dense retrieval
Embed every chunk once at index time, embed the query at request time, and return the k nearest vectors by cosine similarity. One embedding call, one approximate-nearest-neighbor (ANN) lookup, done. Latency is dominated by the query embedding (single-digit to low-tens of milliseconds) plus the ANN search (sub-millisecond to a few ms on an HNSW index).
This is the correct baseline and often the correct product. It wins when your corpus is semantically clean, your queries paraphrase the documents, and you don't need surgical precision. It falls down on exact terms — part numbers, error codes, rare proper nouns — because embeddings smear specific tokens into a general neighborhood. Reach for this first, measure, and only add machinery when the recall numbers tell you to.
2. Hybrid retrieval (BM25 + dense) with fusion
Run a lexical search (BM25 over an inverted index) and a dense search in parallel, then fuse the two ranked lists — Reciprocal Rank Fusion (RRF) is the workhorse because it needs no score calibration, just ranks. BM25 catches the exact-keyword cases dense retrieval fumbles; the dense side catches the paraphrases BM25 misses.
The cost is a second index to build and keep in sync, plus a trivial fusion step. Latency barely moves if the two searches run concurrently. In practice this is the single highest-leverage upgrade over naive retrieval for most real corpora, and I reach for it second, before anything fancier. If you only make one change to a baseline system, make it this one.
3. Cross-encoder reranking
Dense and lexical retrieval are bi-encoders: query and document are embedded independently, so the model never sees them together. A cross-encoder does. You over-retrieve — pull the top 50 or 100 candidates cheaply — then feed each (query, document) pair through a model that scores relevance with full cross-attention, and keep the top handful.
This is the most reliable precision boost available, and it wins whenever "the right answer was in the candidate set but ranked 12th." The cost is real: a cross-encoder pass over 50 candidates adds tens to low-hundreds of milliseconds and a per-candidate compute charge, whether you run a small hosted reranker (Cohere Rerank, Voyage rerank) or self-host a bge-reranker. Rerank a small candidate set, not a large one — cost scales linearly with candidates.
4. Query rewriting and decomposition
The user's literal words are often a bad search query. A short LLM call rewrites the question into a cleaner retrieval query, or decomposes a multi-part question ("compare our 2024 and 2025 refund policies") into separate sub-queries, each retrieved independently. Conversational systems especially need this to resolve pronouns against history before searching.
The tradeoff is a full extra model round trip before you retrieve — a few hundred milliseconds and a few hundred tokens. Use a small, fast model here (Haiku-class) and it stays cheap. It wins on complex, compound, or context-dependent questions and does nothing for simple lookups, so gate it: only rewrite when the query is long, multi-clause, or references prior turns.
5. HyDE (hypothetical document embeddings)
A clever inversion. Instead of embedding the question and searching for documents, you ask an LLM to write a fake answer to the question, then embed that hypothetical document and search with it. The intuition: an answer looks more like the target passage than the question does, so it lands closer in embedding space.
HyDE helps in zero-shot settings and specialized domains where the query vocabulary and the document vocabulary diverge. It costs a generation call per query (the latency of writing a paragraph) and it can hallucinate the search off a cliff if the fake answer is confidently wrong. I treat it as a situational tool, not a default — measure it against plain hybrid retrieval before committing, because it frequently loses.
6. Small-to-big (parent-document) retrieval
Embed and search over small chunks — a sentence or two, which embed with high precision — but return the parent they came from (the full paragraph or section) to the generator. You get the retrieval accuracy of small chunks and the context completeness of large ones, without forcing a single chunk size to serve both jobs.
The cost is bookkeeping: store the child-to-parent mapping and dedupe when several children point at the same parent. Latency is unchanged. This is a low-risk, high-value pattern for structured documents, and I'll take it over fiddling endlessly with a global chunk-size parameter almost every time.
7. Contextual retrieval
Chunks lose their context. A paragraph that says "the limit was raised to 10,000" is useless if you can't tell which limit, which product, which version. Contextual retrieval fixes this at index time: before embedding each chunk, prepend a short LLM-generated blurb situating it in the document ("This is from the Q3 rate-limit policy for the Enterprise tier; it describes..."). You embed the situated version.
The obvious objection is cost — one LLM call per chunk across the whole corpus. But this is a one-time indexing expense, and prompt caching makes it cheap: cache the full document once, then generate context for each of its chunks against the cached prefix at roughly a tenth of the input price. Query-time latency is unchanged. For the money, this is one of the best recall improvements you can buy, and it stacks cleanly with hybrid retrieval and reranking.
8. GraphRAG / knowledge-graph retrieval
For questions that require connecting facts scattered across many documents — "which of our vendors are affected by the same upstream dependency?" — flat chunk retrieval struggles, because no single chunk contains the answer. GraphRAG builds a knowledge graph at index time (extract entities and relationships with an LLM, cluster them into communities, summarize each community) and retrieves over that structure. Global questions get answered from community summaries; specific ones traverse the graph.
This is the heavyweight of the list. The index build is genuinely expensive — many LLM calls to extract and summarize an entire corpus — and the graph needs maintenance as documents change. Query-time cost is moderate. Justify it only when your value is in the connections between facts rather than the facts themselves; for straight lookup it's overkill.
9. Agentic / iterative RAG
Instead of retrieve-then-generate, the model runs a loop: retrieve, reason about whether it has enough, formulate a follow-up search, retrieve again, and only answer when satisfied. Retrieval becomes a tool the model calls, sometimes several times, adapting each query to what the last one turned up.
This handles multi-hop and open-ended research questions that a single retrieval pass can't. It's also the most expensive and highest-variance option: multiple model turns and multiple retrievals per query means unpredictable latency (seconds, not milliseconds) and token cost that scales with how hard the question is. Put a hard cap on iterations, and don't route simple questions here — the cost-per-answer difference between this and naive retrieval can be two orders of magnitude.
10. Long-context "just stuff it in" vs. cache-augmented generation
With million-token context windows, one option is to skip retrieval infrastructure entirely and paste the whole corpus (or a large slice) into every prompt. No vector database, no chunking, no embedding pipeline. The model sees everything and sorts it out.
The catch is that you pay for those input tokens on every call, and quality degrades as relevant facts get buried in the middle of an enormous context. Cache-augmented generation is the sane middle ground: put the stable corpus behind a prompt-caching breakpoint so it's processed once and read back at roughly a tenth of the price on subsequent requests, with only the varying question outside the cache. This wins when the knowledge base is small enough to fit, changes rarely, and is queried repeatedly. Above a few hundred thousand tokens, or with a large corpus where each query needs a different slice, retrieval is still cheaper and sharper.
Where I actually start
Naive dense retrieval to establish a baseline and a metric. Then hybrid plus a reranker, which covers most of the ground most of the time. Contextual retrieval when indexing cost allows, because it's a cheap, compounding win. Everything past that — HyDE, GraphRAG, agentic loops — is a response to a measured failure, not a starting position. RAG isn't a technique you pick; it's a stack you assemble, and the cost curve gets steep fast toward the bottom of this list.