Statistical document similarity in decentralized knowledge repositories
Knowledge workers frequently iterate on documentation across separate files: drafting project postmortems, outlining client proposals, or duplicating standard operating procedures across team directories. Unlike byte-for-byte duplicates, these documents evolve independently. One version incorporates supplementary corrective actions, while another updates meeting attendee rolls or reorganizes paragraph sequences.
Standard filesystem deduplication tools rely strictly on hash equality and overlook these partial duplicates entirely. The Near-Duplicate Note Reviewer fills this gap by performing token-level statistical similarity audits directly in client-side memory.
Mathematical foundation: word-level bigram shingling and Jaccard metrics
To quantify textual overlap without requiring computationally prohibitive language models, the engine leverages set-theoretic shingling:
- Structural Tokenization: Frontmatter headers, code fences, and Markdown markup characters are stripped to expose clean lexical prose tokens.
- Bigram Shingle Generation: Adjacent word pairs (such as
incident overview,latency exceeding,routing tier) are mapped into unique token sets for each document. - Jaccard Coefficient Calculation: The intersection size of shared bigrams is divided by their total union size. This yields a transparent coefficient from 0% to 100% that accurately reflects shared prose patterns regardless of formatting differences.
The short-document paradox: guarding against false positive alerts
A frequent pitfall of automated text comparison algorithms is the short-document paradox. Brief notes—such as five-line daily standup logs or shopping checklists—contain a very limited vocabulary. When compared, two brief notes might mathematically exhibit a 60% similarity score simply because both list items like "Review tickets" and "Update team".
Recommending deletion for such notes would be disastrous. Our engine implements an explicit brevity guard: any document pair where either note contains fewer than 15 words is automatically classified as a Short Generic Note. The system warns the user that lexical similarity cannot guarantee factual redundancy and explicitly refrains from staging either file.
Transparent evidence extraction for human-in-the-loop review
Automated deduplication should never operate as an opaque black box. For every candidate pair discovered, the engine extracts the specific sentences exhibiting the highest concentration of shared bigram shingles. Presenting side-by-side excerpts enables you to verify the substance of the overlap in seconds before deciding whether to consolidate documents or retain both as distinct historical records.
Complementary private text engineering tools
Explore companion utilities designed for rigorous local knowledge base curation:
- Exact Duplicate Note Finder: Cryptographic and normalized detection of identical Markdown duplicates.
- Unused Attachment Reviewer: Detect orphaned media files and reclaim vault storage.
- Vault Health Inspector: Comprehensive audit of note backlinks, orphan attachments, and file integrity.
- All Tools Directory: Discover private browser-based utilities for Markdown workflows.