Skip to content

Wikantik

Systems engineering blueprint contrasting torn in-place file writes against hardened POSIX atomic file lifecycles

Silent Data Loss and Destructive Admin Tools: Fixing Storage Invariants

From Wikantik · 22 articles · latest

Systems engineering blueprint contrasting torn in-place file writes against hardened POSIX atomic file lifecycles
Storage invariants: Replacing unsafe in-place file overwrites and cascading admin deletes with POSIX atomic writes and diff-aware index synchronization.

During the October 22, 2026 storage profiling campaign, low-level Linux syscall tracing (strace) and code review identified two significant data integrity defects in Wikantik: silent data loss during page saves and an administrative index rebuild routine that executed cascading deletes across relational embedding tables.

These behaviors were not introduced by modern agentic features; they were legacy implementations inherited from the Apache JSPWiki foundation that had persisted through earlier framework ports.

This postmortem analyzes the failure mechanisms of both defects and documents the POSIX-level storage hardening deployed in milestone 2.4.62.

Defect 1: In-Place Truncation and Exception Suppression in PrintWriter

When an author or automated agent saves a wiki page, the durability contract requires that the content is safely flushed to non-volatile storage or an explicit error is returned.

Tracing AbstractFileProvider.java under strace captured the following syscall sequence:

// Legacy in-place overwrite pattern observed in strace:
openat(AT_FDCWD, "/var/wikantik/pages/Main.md", O_WRONLY|O_CREAT|O_TRUNC, 0664) = 43
write(43, "...", 4096) = 4096
close(43) = 0

This sequence presented three specific structural vulnerabilities:

  1. In-Place Truncation (O_TRUNC): Opening the existing file with O_TRUNC instantly reduced the file size to zero bytes before new content was written. If an unexpected power cut, kernel panic, or container kill occurred during the write, the file was left empty, destroying the previous page content.
  2. Suppressed Write IOExceptions: Writing occurred via standard Java PrintWriter. According to the Java SE specification, PrintWriter methods catch and suppress all underlying IOException instances, setting an internal error flag that callers rarely inspect. If disk space became exhausted mid-write (ENOSPC), the failure was not thrown.
  3. Swallowed Provider Exceptions: The outer layer in FileSystemProvider.java caught exceptions, emitted a single un-stratified log warning, and returned normally. The web tier and MCP tool responded with HTTP 200 success even if the write failed on the filesystem.
Remediation: POSIX Atomic Write Lifecycle

In commits 18319d5126 and 5c5c818f43, page persistence was refactored to enforce atomic write semantics:
1. Write to a unique sibling temporary file in the same directory (.page.md.tmp-UUID) using an unbuffered OutputStreamWriter that throws on I/O errors.
2. Force bytes to disk with FileChannel.force(true), issuing an explicit fsync on the file descriptor.
3. Perform an atomic filesystem rename via Files.move(..., ATOMIC_MOVE).
4. Call fsync on the parent directory channel to commit directory entry modifications.
5. Bubble any failure up as an explicit ProviderException, failing the save request cleanly.

Defect 2: Destructive Cascades in rebuild-indexes

The second issue was located in POST /admin/content/rebuild-indexes, an administrative endpoint intended to refresh search indexes following bulk imports.

Inspection of ContentIndexRebuildService.java revealed that the rebuild procedure began by deleting all rows in the content chunk table:

// ContentIndexRebuildService.java:317 (v2.4.61)
chunkRepo.deleteAll();

In the PostgreSQL schema, foreign keys from child tables were configured with ON DELETE CASCADE. Executing deleteAll() produced severe operational side effects:

  • Cascading Deletes: Deleting all 18,635 rows in kg_content_chunks triggered cascading deletions across content_chunk_embeddings, chunk_entity_mentions, and experiment_embeddings, generating over 149,000 Write-Ahead Log (WAL) entries.
  • Loss of Vector Index Availability: All 18,627 dense embeddings were erased. The rebuild service then attempted to generate embeddings sequentially through Ollama. On hosts without dedicated GPU inference, embedding batches timed out, leaving vector search queries empty for hours.
  • Permanent Deletion of Extracted Mentions: Entity mentions in chunk_entity_mentions are generated by an offline LLM extraction job (wikantik-extract-cli) rather than on-the-fly indexing. Cascading deletes permanently severed all extracted Knowledge Graph mentions from corpus chunks.

Diff-Aware Reconciliation Repair

In commits 1e92107cdf and f114fe11eb, deleteAll() was removed.

The service was rewritten to use ChunkDiff.apply. The rebuild process hashes document sections, compares them to existing database records, and updates only modified or newly created chunks. Unchanged chunks, embeddings, and entity mentions remain untouched. Only un-embedded or modified chunks are queued for vector inference, reducing index refresh time to 15 seconds without service interruption.

Defect 3: Asymmetrical Cache Expiration in Page Listings

The profiling campaign also clarified a cache inconsistency issue: occasionally, newly created or renamed pages failed to appear in navigation listings.

The root cause was asymmetric cache time-to-live settings: getAllPages() cached the complete page collection list for 300 seconds, while individual page metadata items expired after 60 seconds. When an individual entry expired from the cache while the parent collection remained cached, getAllPages() skipped the missing item during iteration, returning an incomplete page listing.

In commit b50c63166d, getAllPages() was updated to query an immutable point-in-time snapshot, preventing transient cache misses from altering directory output.

Systems Engineering Takeaway

Modernizing enterprise software requires validating low-level storage invariants against operating system APIs. Assumptions about file persistence must be verified through raw syscall observation, ensuring that crash recovery, atomic file renaming, and error propagation operate deterministically.

The next article analyzes production operations under autonomous AI agents: examining thread lifecycle leaks in the Model Context Protocol SDK, token budgeting, and the ongoing operational costs of metadata curation.

References and Source Code Truth

Keep reading

Comments