Skip to content

Wikantik

Technical architectural schematic of hybrid retrieval merging Lucene BM25 keyword search and pgvector dense semantic embeddings via Reciprocal Rank Fusion

Building the Hybrid Retrieval Engine: BM25 Meets pgvector and Reciprocal Rank Fusion

From Wikantik · 22 articles · latest

Technical architectural schematic of hybrid retrieval merging Lucene BM25 keyword search and pgvector dense semantic embeddings via Reciprocal Rank Fusion
The hybrid retrieval architecture: Combining the precision of Lucene BM25 lexical search with the semantic understanding of pgvector embeddings via Reciprocal Rank Fusion.

When engineering retrieval for technical knowledge bases, developers frequently fall into a false dichotomy: stick with classic inverted index search (Lucene / Elasticsearch BM25) or replace everything with a dense vector database. In practice, both approaches fail on realistic engineering queries when used in isolation.

Pure vector search excels at conceptual intent ("how do we handle user authentication?") but degrades rapidly on exact keyword matches, domain acronyms, function names, and error codes. Conversely, pure BM25 lexical search handles exact terms flawlessly but cannot connect semantically equivalent concepts across differing vocabularies.

To handle both exact symbol lookups and conceptual queries, Wikantik combines Apache Lucene BM25 with PostgreSQL pgvector embeddings, merging results through Reciprocal Rank Fusion (RRF) over heading-aware document chunks.

The Anatomy of Modern Content Chunking

Vector search is only as good as the underlying chunks it embeds. In early experiments, naive fixed-window token slicing (e.g., rigid 512-token chunks with 50-token overlap) generated severe semantic truncation: cutting code examples in half, severing markdown table headers, and producing orphan sentences without context.

Wikantik engineered a purpose-built ContentChunker designed specifically for structured technical markdown:

  • Heading Hierarchy Tracking: Every chunk maintains an explicit heading_path (e.g., Architecture > Storage Tier > Caching). When chunks are embedded and retrieved, the heading path provides necessary topical anchors even when the chunk itself is brief.
  • Atomic Block Preservation: Fenced code blocks, markdown tables, and definition lists are treated as atomic units that are never bisected by chunk boundaries.
  • The Merge-Forward Algorithm: Micro-sections (such as one-line "Overview" or "See Also" headers) create noise in vector indexes. Wikantik raised its mergeForwardTokens threshold from 8 to 150 tokens. Sibling sections below this threshold are accumulated and merged forward into subsequent blocks.

Across the production corpus, total chunk count dropped from 39,264 to 23,256 rows (−41%), while mean chunk size increased from 103 to 174 tokens (+69%). This reduced embedding generation time and table storage by roughly 40% without reducing retrieval recall.

Dense Vector Indexing with pgvector and Ollama

Rather than introducing a dedicated vector database service requiring separate replication, backups, and network latency, Wikantik integrated dense vector storage directly into PostgreSQL using the pgvector extension.

The data pipeline operates asynchronously on page saves:

  1. When a page is saved, a ChunkProjector page filter decomposes the markdown into structured chunks and computes content hashes.
  2. An asynchronous listener detects dirty chunks and submits them in micro-batches to an embedding client (utilizing local inference via Ollama with qwen3-embedding-0.6b).
  3. The resulting 1024-dimensional vectors are stored in content_chunk_embeddings, indexed via Hierarchical Navigable Small World (HNSW) graphs with cosine distance metrics.
Production Reliability: The Query Embedder Circuit Breaker

External inference endpoints can lag or fail during high concurrency. To prevent user queries or agent calls from hanging, Wikantik wraps its QueryEmbedder with a hand-rolled circuit breaker (CLOSED, OPEN, HALF_OPEN) and a Caffeine in-memory cache. If embedding response times exceed 1.5 seconds or the service throws consecutive timeouts, the circuit trips to OPEN. The retrieval engine immediately fails-closed to fast Lucene BM25 keyword search, ensuring the wiki never stops responding.

Reciprocal Rank Fusion (RRF) and Page Aggregation

Once candidate result lists are generated from both engines—Lucene BM25 producing lexical scores and pgvector producing cosine similarities—the challenge becomes fair fusion. Because BM25 scores (unbounded positive floats) and cosine similarities (0.0 to 1.0) cannot be normalized reliably across differing query lengths, Wikantik uses Reciprocal Rank Fusion (RRF):

RRF_Score(d) = Σ ( w_k / ( k_rrf + rank_k(d) ) )

Where k_rrf = 60 dampens the influence of top ranks, and w_k provides tunable weights between lexical and dense signals.

Because queries ultimately need to resolve to whole pages or relevant contributing sections, the individual chunk scores are folded into parent pages using a SUM_TOP_3 aggregation algorithm: a page's final score is determined by the sum of its three strongest contributing chunks. This prevents massive reference pages with dozens of weak mentions from outranking focused, highly relevant architectural articles.

The Production Result

In our evaluation harness comparing pure BM25, pure dense embeddings, and hybrid fusion across technical queries, the combined pipeline outperformed either method alone:

  • Pure BM25: Excelled on exact terms (recall@5 of 0.68) but failed on conceptual rephrasing (recall@5 of 0.42).
  • Pure Dense: Captured conceptual intent (recall@5 of 0.74) but frequently missed specific class names and acronyms.
  • Hybrid RRF (Dense-Weighted): Achieved a combined recall@5 of 0.89, successfully finding target technical documentation across both query modes.

With hybrid chunk retrieval in place, we next tested whether adding a Knowledge Graph reranking stage would improve recall further. For the current production parameters, mathematical formulation, and bundle pipeline reference, see Wikantik Hybrid Search and Context Bundle Architecture on the wiki.

References and Source Truth

Keep reading

Comments