Skip to content

Wikantik

Engineering postmortem diagram comparing complex multi-hop Knowledge Graph traversal against empirical benchmark curves showing zero retrieval lift

Evaluating GraphRAG: Benchmarks and the Removal of Knowledge Graph Reranking

From Wikantik · 22 articles · latest

Engineering postmortem diagram comparing complex multi-hop Knowledge Graph traversal against empirical benchmark curves showing zero retrieval lift
The GraphRAG postmortem: Why multi-hop entity traversal and co-mention graph reranking added operational latency and zero empirical retrieval lift compared to clean dense-chunk hybrid search.

A common hypothesis in retrieval-augmented generation (RAG) is that combining vector search with an extracted Knowledge Graph—often referred to as "GraphRAG"—will recover multi-hop context that flat embeddings miss.

In early 2026, we implemented this approach in Wikantik. We added relational tables (kg_entities, kg_relations, kg_entity_mentions), built an asynchronous entity extraction pipeline, and integrated a graph proximity reranker into the search pipeline.

When we benchmarked the graph reranker against our baseline hybrid search (Lucene BM25 + pgvector) across 40 technical queries, the graph stage produced zero net lift at Recall@12 and a slight regression at Recall@5 while adding 120ms–280ms of latency per query. In commit f39dc0402f, we removed the 2,464-line graph reranking subsystem.

This post covers the design of that subsystem, the benchmark results, and why page-level entity co-mention boosting underperformed chunk-level hybrid search.

The Hypothesis: Traversing Entity Relationships

The motivation for GraphRAG arose from real limitations in dense retrieval. When an engineer asks a complex relational question—such as "How does our session failover interact with distributed cache eviction policies?"—dense embeddings often struggle if no single document explicitly collocates both concepts in a single paragraph.

The theoretical solution was a four-stage retrieval pipeline:

  1. BM25 Lexical Retrieval: Fetch candidates matching exact keywords.
  2. Dense Vector Retrieval: Fetch candidates matching semantic embeddings.
  3. Reciprocal Rank Fusion (RRF): Merge lexical and semantic candidate lists.
  4. Knowledge Graph Reranking: Resolve entities mentioned in the query, find their 1-hop and 2-hop neighbors in an in-memory graph index, and boost the rank of candidate pages that share co-mention density or relational edges with the query entities.

Implementation Architecture

To test this approach in production, we built four components:

  • GraphProjector: An asynchronous event listener that scanned page saves and extracted structured entity triples.
  • QueryEntityResolver: A fast parser that mapped incoming user query terms to known entity nodes in the database.
  • InMemoryGraphNeighborIndex: A low-latency graph cache holding entity adjacency matrices in memory to avoid crushing PostgreSQL with recursive SQL joins during user searches.
  • GraphProximityScorer & GraphRerankStep: A scoring engine that applied decaying proximity boosts based on shortest-path graph distance and relationship confidence scores.

We tuned extraction prompts, tested local LLMs, and even generated a gold-standard benchmark slice using Claude to ensure extraction errors weren't bottlenecking the graph.

Architectural Insight: The Flawed Assumption

The fundamental flaw in GraphRAG for contextual retrieval is a category error: entity proximity is not section relevance. A Knowledge Graph accurately records that two concepts are structurally related in the domain ontology. It has zero awareness of which paragraph in a 3,000-word architectural document actually answers the engineer's specific operational question.

Benchmark Results (eval/kg-spike)

In June 2026, we ran the Ceiling Spike (eval/kg-spike) evaluation harness across 40 technical queries against the documentation corpus, comparing baseline hybrid search against hybrid search with graph reranking:

  • Recall@12: Exactly zero net lift (0.00). The graph reranker did not bring a single relevant missing document into the top-12 context window that wasn't already captured by dense-chunk vector search and BM25.
  • Recall@5: A net regression of −1. The graph rerank actually displaced highly specific, laser-focused technical sections in favor of broad, high-degree "hub" pages that contained high entity counts but negligible answering text.
  • Latency Penalty: Entity resolution and graph adjacency lookups added 120ms to 280ms of overhead to every query.
  • Operational Debt: Entity extraction at write time created massive queue backlogs, table bloat in PostgreSQL, and required continuous background reconciliation.

The Diagnosis: Why the Graph Failed

Analyzing the evaluation failures revealed two decisive structural reasons why Knowledge Graph reranking collapsed in practice:

1. High-Degree Hub Distortion

In any authentic technical documentation corpus, certain foundational concepts (e.g., "Authentication", "HTTP Servlet", "PostgreSQL") act as high-degree graph hubs with hundreds of edges. The graph proximity algorithm constantly boosted these generic hub documents into the top ranks, pushing down the specific, nuanced implementation guide the user actually needed.

2. The Granularity Mismatch

Modern LLMs do not consume whole pages; they consume targeted chunks within tight token budgets. A page-level Knowledge Graph operates at the wrong level of abstraction. A page might have 84 entity mentions across 10 subsections, but only Subsection 4.2 contains the solution to the query. Boosting the entire page based on aggregate entity density produced noisy context bundles that wasted LLM context windows.

Removing the Graph Reranker

Based on the evaluation metrics, we removed the graph reranking subsystem on July 30, 2026 in commit f39dc0402fe:

  • Removed GraphRerankStep, GraphProximityScorer, QueryEntityResolver, PageMentionsLoader, and InMemoryGraphNeighborIndex.
  • Removed 2,464 lines of Java code and unit tests.
  • Deprecated the wikantik.search.graph.* configuration namespace.

Our actual retrieval gains came from four adjustments to the chunking and ranking pipeline:

  1. Heading-Aware Chunking: Preserving markdown header hierarchies so every vector chunk knows its structural home.
  2. Atomic Blocks: Never bisecting code blocks or tables.
  3. Merge-Forward Sizing: Keeping chunk sizes around 174 tokens to give embedding models sufficient context.
  4. Reciprocal Rank Fusion (RRF): Merging chunk-level BM25 and dense vectors.

Those four adjustments improved section recall from 0.60 to 0.74 without graph extraction overhead and kept query latency under 20ms.

The True Place for Graphs in Wikantik

Abandoning graph rerank did not mean abandoning graphs altogether. Instead, it forced an essential architectural clarification: The Knowledge Graph is a curated navigation and semantic query surface for humans and agents, not a retrieval ranking hack.

In the next post, we examine how we separated our graph model into two distinct subsystems: the Page Graph of document hyperlinks and the Knowledge Graph of RDF/OWL semantic triples.

References and Source Truth

Keep reading

Comments