Skip to content

Wikantik

Why We Clobbered the Knowledge Graph: Search Ranking Realities and Architecture Simplification in Wikantik

From Wikantik · 22 articles · latest

Systems architecture blueprint illustrating the removal of an extracted entity knowledge graph in favor of high-density vector embeddings and a deterministic page link graph
Architectural simplification in Wikantik 2.5: Pruning 70,500 lines of heuristic entity extraction and 15 relational tables to focus the retrieval pipeline on dense vector embeddings and deterministic page structures.

Between releases 2.5.12 and 2.5.14, we permanently removed the Knowledge Graph from Wikantik. The deletion was sweeping: 699 files modified, 15 PostgreSQL tables dropped in Flyway migration V070, 21 ontology relationship predicates retired, and a net reduction of 70,516 lines of Java and TypeScript code. Every accompanying background service—the automated LLM entity extractor, the multi-agent proposal judge, the inclusion policy engine, hub discovery, SHACL validation pipelines, and 15 Model Context Protocol (MCP) tools—has been excised from the repository.

In modern software engineering, deleting a major subsystem that required months of development is rarely done lightly. The rationale for this architectural decision was straightforward and empirical: the Knowledge Graph was not improving search ranking or retrieval quality. Despite its conceptual elegance on paper, the extracted entity graph generated substantial operational latency, compounding synchronization complexity, and noisy traversal paths that consistently underperformed simpler, deterministic retrieval primitives. When an architectural component fails to deliver measurable retrieval gains while imposing an ongoing maintenance tax, the correct engineering choice is to clobber it.

The Theoretical Promise vs. Production Reality

The original design thesis behind Wikantik's Knowledge Graph was inspired by the broader industry enthusiasm for “GraphRAG” in 2024. The hypothesis held that while dense vector embeddings capture fuzzy semantic proximity across text chunks, they struggle with precise relational reasoning (such as identifying all components dependent on a specific subsystem, or tracing multi-hop corporate hierarchies). By introducing an automated pipeline to extract named entities (wk:Person, wk:Organization, wk:SoftwareSystem) and directed predicates (wk:dependsOn, wk:maintains, wk:authorizes), the system was expected to build a queryable semantic mesh that would elevate search relevance and allow LLM agents to execute multi-hop reasoning over the wiki corpus.

To execute this thesis, Wikantik built a comprehensive semantic web infrastructure:

  • 15 Dedicated Relational Tables: Staging tables for raw extracted mentions (chunk_entity_mentions), candidate node deduplication (kg_nodes), directional relations (kg_edges), an asynchronous curation workflow (kg_proposals), and graph clustering partitions (hub_members, hub_proposals).
  • LLM Extraction & Evaluation Pipeline: Asynchronous batch workers querying local Ollama or cloud models to parse document chunks into entity-relation triplets, followed by a secondary LLM “judge” tasked with scoring proposal validity and rejecting hallucinations.
  • Ontology Governance: Strict SHACL (Shapes Constraint Language) validation rules enforcing type safety, domain/range constraints, and preventing ontological drift across Apache Jena in-memory models.

In controlled synthetic benchmarks with tiny toy datasets, this architecture appeared promising. In live production against a diverse, evolving corpus of technical documentation, architectural decision records, and operational runbooks, the cracks immediately widened into systemic failures.

Failure Mode 1: Entity Resolution and the Graph Noise Explosion

The primary algorithmic failure was entity resolution. Natural language documentation contains countless orthographic and contextual variants for the exact same underlying concept. A systems engineer might refer to “PostgreSQL” on one page, “Postgres DB” on another, “the relational datastore” in an ADR, and pg_dump in an operational script.

Because the extractor operated on local document chunks, it continually synthesized disparate entity nodes for identical targets. Attempting to solve this through automated string distance (Levenshtein, Jaro-Winkler) or secondary embedding clustering introduced catastrophic false positives: distinct concepts were erroneously merged, while identical systems remained fractured into disconnected subgraphs.

The resulting graph was simultaneously sparse where density was needed (missing true architectural links) and dense where noise was toxic (cross-linking unrelated concepts because an LLM hallucinated a generic wk:relatesTo predicate). When automated agents traversed these edges during multi-hop retrieval, they were routinely led down semantic detours that polluted the final context window with completely irrelevant text.

Failure Mode 2: Zero Empirical Uplift in Search Ranking

To evaluate whether the Knowledge Graph justified its presence, we instrumented systematic retrieval quality evaluations across our corpus, comparing three retrieval strategies:

  1. Dense Vector Retrieval: Cosine similarity against 768-dimensional chunk embeddings indexed via pgvector using Hierarchical Navigable Small World (HNSW) indexes.
  2. Page Graph Determinism: Ranking pages based on authored bidirectional wikilinks ([PageName]), parent-child namespaces, and explicit citation structures.
  3. GraphRAG Traversal: Expanding query candidates across 1-hop and 2-hop extracted entity paths weighted by graph centrality (PageRank on kg_edges).
Retrieval PipelineMean Reciprocal Rank (MRR)NDCG@10P95 Query LatencyIngestion Write Overhead
Dense pgvector + BM25 (Hybrid)0.8420.88742 ms< 350 ms / page
Deterministic Page Graph Only0.7810.81412 ms< 20 ms / page
Knowledge Graph Traversal (1-Hop)0.6940.723285 ms4,200 ms / page
Hybrid + Knowledge Graph Expansion0.8180.841340 ms4,200 ms / page

The empirical results were damning. Not only did Knowledge Graph traversal fail to outperform hybrid dense retrieval; activating graph expansion actually reduced Mean Reciprocal Rank from 0.842 down to 0.818. Spurious edges introduced irrelevant context chunks into the top-10 result set, diluting context density for downstream reasoning models. Furthermore, running multi-hop relational joins across relational tables ballooned query latency by more than 700%.

Architectural Principle (The Complexity Rent Test):

Every architectural subsystem must pay rent in the form of measurable user value, reduced operational cost, or superior output quality. If an automated subsystem consumes 80% of ingestion processing time, burns millions of inference tokens, and introduces 15 database tables, but yields a negative delta on retrieval accuracy, it must be removed without nostalgia.

Failure Mode 3: The Operational Ingestion Tax

Beyond search ranking failure, the operational burden of maintaining the Knowledge Graph was crippling to system throughput:

  • Ingestion Wall-Clock Bottlenecks: When an engineer or autonomous agent edited a 2,000-word page, generating vector embeddings took roughly 300 milliseconds. In contrast, running the chunk extractor, generating candidate entity triplets, executing the LLM judge, and validating SHACL constraints took between 4 and 8 seconds per page. During batch document imports, the extraction worker queue backed up for hours.
  • Database Lock Contention & Migration Risk: Maintaining relational foreign keys across 15 interconnected tables caused frequent row locks during high-concurrency automated agent runs. Upgrading schema definitions required complex multi-step data migrations that risked data integrity during crash recovery.
  • Cognitive Overhead for Operators: Wiki administrators were forced to navigate proposal curation queues, review rejected relationship types, and manage vocabulary drift across an ontology that provided no tangible benefit to readers.

The Execution: How We Clobbered the Subsystem

To safely decommission the Knowledge Graph without risking data corruption or service downtime, we executed a phased two-step removal across releases 2.5.12 and 2.5.13:

Step 1 (Release 2.5.12): Fencing and Dark-Launch Retirement

In commit 7c99136a08, the configuration property wikantik.knowledge.enabled was switched to default to false. When disabled, the entire ingestion pipeline bypassed entity extraction, and ontology rebuilds were restricted to document classes (wk:Article) and SKOS concept schemes.

All reader-facing /knowledge-graph UI routes and admin curation panels were stripped from the web application. Simultaneously, an ArchUnit architectural rule (KnowledgeGraphFenceArchTest) was frozen in CI/CD, guaranteeing that no existing or newly written code in the core wiki runtime could introduce new dependencies on the quarantined knowledge graph packages.

Step 2 (Release 2.5.13): Total Physical Deletion

With production running smoothly for multiple days with zero reads against the graph tables, we executed the definitive deletion pass in commit 1b7ec8b6cc:

  1. Database Table Archival & Drop: We executed a full pg_dump of the 15 legacy graph tables to secure cold backup storage. Flyway migration V070__drop_knowledge_graph.sql then executed DROP TABLE cascades against kg_nodes, kg_edges, kg_proposals, chunk_entity_mentions, and all associated hub tables.
  2. Codebase Purge: We deleted all 154 production Java files and 106 test classes associated with extraction, curation, judging, and graph REST resources. All bin/kg-* administrative shell scripts were removed from disk.
  3. Ontology Trimming: The ontology was streamlined strictly to document structural classes and topic concepts. The 9 entity classes, 21 relationship predicates, and SHACL constraint checkers were dropped, leaving get_ontology fast, deterministic, and lightweight.

What Remains: High-Density Primitives That Actually Work

Crucially, retiring the Knowledge Graph did not degrade Wikantik's semantic capabilities. In fact, retrieval is now substantially faster, cleaner, and more accurate because the system focuses exclusively on high-signal primitives:

  • Dense Chunk Retrieval: Tables kg_content_chunks and content_chunk_embeddings remain fully active. Text chunks are embedded via lightweight local or cloud embedding models into pgvector, delivering sub-50ms semantic search that handles phrasing nuances, synonyms, and user intent effortlessly.
  • The Deterministic Page Graph: The true knowledge graph of an organization is not synthesized by an LLM guessing at triplets; it is authored deliberately by humans and agents through explicit internal links ([PageName]), category tags, and formal citations. Wikantik's Page Graph remains an instant-access, deterministic structural map that accurately represents document relationships without hallucinations—see Page Graph vs. Knowledge Graph and the Hybrid Search Architecture reference on the wiki.
  • Streamlined Agent MCP Context: Autonomous coding agents interacting with Wikantik via the Model Context Protocol now query retrieve_context, receiving high-relevance, deduplicated text bundles and citations without wasting tokens navigating synthetic relationship graphs.

Conclusion: The Value of Negative Work

In software engineering, there is a natural bias toward addition—adding new models, adding layers of abstraction, adding graph databases, and accumulating architectural weight under the assumption that more complex pipelines produce smarter systems. Wikantik 2.5 stands as a testament to the opposite discipline.

By removing 70,500 lines of unproven complexity, dropping 15 database tables, and cutting ingestion latency by 90%, we made Wikantik significantly faster, dramatically more reliable, and mathematically more accurate. The future of agent-grade knowledge retrieval belongs to fast, dense embeddings anchored in deterministic human structure—not brittle, synthetic web graphs.

Authoritative Outbound References

Keep reading

Comments