Why GraphRAG turns knowledge into a network, not a pile of text

Why GraphRAG turns knowledge into a network, not a pile of text

Ask an LLM to summarize what your whole document corpus is about and you'll quickly hit a wall. Plain retrieval-augmented generation (RAG) is brilliant at finding a needle in a haystack, but it fails when the question requires connecting the dots across the entire haystack. That is exactly the blind spot Microsoft's GraphRAG paper was written to fix.

"RAG fails on global questions directed at an entire text corpus, such as 'What are the main themes in the dataset?', since this is inherently a query-focused summarization (QFS) task, rather than an explicit retrieval task." — Edge et al., From Local to Global: A Graph RAG Approach to Query-Focused Summarization (arXiv:2404.16130)

This article digs into why that failure happens, exactly how GraphRAG fixes it, and — most usefully — when each retrieval approach (plain vector RAG, GraphRAG, or a hybrid) is the right tool, backed by verifiable sources.

First, a refresher: how plain RAG actually works

Before we talk about graphs, it's worth being precise about what "plain RAG" does under the hood, because every failure mode we discuss comes from one of these steps. RAG (retrieval-augmented generation) was formalized by Lewis et al. in 2020 — the core idea is to augment the LLM with retrieved context instead of relying only on its parametric memory. A typical RAG pipeline has six steps:

  1. Chunking. The corpus is split into text chunks. The chunk size is a real design decision, not a detail: in GraphRAG's own ablation (Section 3.1.1), GPT-4 extracted almost twice as many entity references from 600-token chunks as from 2400-token chunks on the HotPotQA dataset — larger chunks degrade the recall of information early in the chunk. (Source: arXiv:2404.16130)
  2. Embedding. Each chunk is turned into a high-dimensional vector using an embedding model. Two popular families:
  3. Vector database. The chunk embeddings are stored in a vector store (Pinecone, Weaviate, Qdrant, pgvector, FAISS, etc.). At query time the user's question is embedded with the same model and the store returns the K nearest neighbors by cosine similarity.
  4. Retriever. The vector store returns the top-K most-similar chunks. This is the step that "loses" when a question needs entities joined across chunks.
  5. Re-ranking (optional). A second, more expensive model re-sorts the top-K hits to surface the most relevant ones before they reach the LLM.
  6. Generator. The retrieved chunks are stuffed into the LLM's context window and it writes the answer.
A typical RAG pipeline: chunk → embed → retrieve → re-rank → generate.
A typical RAG pipeline: chunk → embed → retrieve → re-rank → generate.

The critical limitation: step 4 is a best-first similarity search. It returns chunks that look like the question. It has no idea which chunks relate to each other. When the answer lives at the intersection of many documents, no single chunk is a good "nearest neighbor," so the top-k retrieval returns a handful of loosely related fragments — and the LLM has to "connect the dots" that were never handed to it.

That's the real reason the GraphRAG authors wrote that opening quote. Vector RAG is excellent at explicit fact retrieval, and it's precisely bad at the thing the paper calls global sensemaking — "what are the main themes in this 1-million-token dataset?"

The fix: build a graph, then summarize the graph

GraphRAG's core insight is to re-index the corpus into a knowledge graph first, and then make the graph itself the retrieval substrate. The paper's pipeline (Figure 1) has two phases — indexing time and query time:

Indexing (one-time, offline):

GraphRAG indexing: chunks → entities/relations → knowledge graph → Leiden communities → recursive community summaries.
GraphRAG indexing: chunks → entities/relations → knowledge graph → Leiden communities → recursive community summaries.
  1. Text Chunks → Entities & Relationships. The LLM is prompted to extract important entities and the relationships between them from each chunk, plus short descriptions. Crucially, the paper's graph index spans nodes (entities), edges (relationships), and covariates (claims).
  2. Entities & Relationships → Knowledge Graph. These triples are merged into a graph. This is where you get the (Florian)-[:OWNS]->(NeuralNine)-style triples: a subject, a predicate, an object.
  3. Knowledge Graph → Graph Communities. The graph is partitioned using community detection — specifically the Leiden algorithm (Traag et al., 2019) — which exploits the graph's inherent modularity to find groups of closely related nodes. This is the step that makes "thematic" structure explicit: related entities get grouped into a community.
  4. Graph Communities → Community Summaries. Each community is summarized by the LLM, recursively bottom-up: summaries at higher levels of the community hierarchy incorporate lower-level summaries. The result is a set of community summaries describing "global descriptions and insights over the corpus."

Query time (map-reduce):

GraphRAG query time: map each community summary to a partial answer, then reduce into the final global answer.
GraphRAG query time: map each community summary to a partial answer, then reduce into the final global answer.
  1. Community Summaries → Community Answers → Global Answer. The paper describes this as map-reduce processing: in the map step, each community summary independently and in parallel generates a partial answer to the query; in the reduce step, all partial answers are combined and summarized into the final global answer. Local search, by contrast, acts directly on specific entities and relationships relevant to the query.

(All of the above is drawn directly from the paper, arXiv:2404.16130, Sections 3.1.1–3.1.6.)

"Comprehensiveness and diversity" — the actual evaluation

The paper evaluates GraphRAG against a conventional RAG baseline (which they call "SS" / "source-and-summarize") plus other configurations, on global sensemaking questions over datasets in the 1-million-token range. The headline metrics are three, each judged by an LLM:

The headline result: GraphRAG leads to substantial improvements over a conventional RAG baseline on both comprehensiveness and diversity. (The paper's Table: GraphRAG beats the baseline with large, statistically significant win-rates — e.g., >70–83% on comprehensiveness across hierarchy levels, p < 0.001 — vs baseline SSRAG's single-digit-to-mid-20s percentages.)

There's also a delightfully counterintuitive finding worth remembering: the authors tested context windows of 8k/16k/32k/64k for the baseline and found the smallest window (8k) was universally best for comprehensiveness (58.1% avg win rate) — consistent with the "lost in the middle" problem that longer contexts dilute the signal. (Source: arXiv:2404.16130, Appendix C.)

Practical note on cost: GraphRAG indexing is genuinely expensive — an LLM is called on every chunk to extract entities, then again to summarize every community, then again per community at query time. Microsoft's own repo warns: "GraphRAG indexing can be an expensive operation… and start small." That cost is the single biggest reason teams hesitate to adopt it — and the reason the alternatives below exist.

Practical takeaways: which retrieval wins, when

Here's the honest, decision-ready summary. The "best" approach depends on the shape of the question you actually need to answer:

Match the retrieval strategy to the shape of the question.
Match the retrieval strategy to the shape of the question.

The bottom line

GraphRAG doesn't replace vector RAG — it complements it. Vector RAG is your precise scalpel for fact and similarity questions; GraphRAG is the wide-angle lens for understanding whole corpora; LightRAG, LazyGraphRAG, and Text2Cypher are the pragmatic tools when you want graph-like structure without GraphRAG's indexing tax. The good engineer's move is not to pick a single paradigm but to match the retrieval strategy to the generality of the question and the structure of the data.

Sources