Content hashing and collision detection across text archives
As personal knowledge graphs expand, redundant documents accumulate across disparate subdirectories. Multi-device background synchronizers (such as Syncthing, iCloud Drive, Nextcloud, and Git merges) frequently spawn duplicate markdown files following connection timeouts or merge conflicts. Similarly, manual project backups, duplicated templates, and clippers leave identical copies scattered across nested folders.
These redundant files degrade search precision, cause wikilinks to resolve unpredictably to stale copies, and increase repository size.
Hashing mechanics: Fowler-Noll-Vo digests and byte metrics
To evaluate hundreds of documents instantaneously without sending private records to an external cloud server, the engine computes deterministic 32-bit FNV-1a (Fowler-Noll-Vo) digests paired with exact byte-length checks. This multi-attribute fingerprinting maps incoming text files into distinct hash buckets with sub-millisecond execution times.
Comparison specifications: exact binary vs normalized semantics
Depending on your workflow, text duplication presents itself in different forms. This utility equips you with three tailored comparison profiles:
- Exact Byte Comparison: Requires character-for-character binary identity. Every byte, line ending, and whitespace character must align perfectly. This is the strictest mode for detecting true binary duplicates.
- Normalized Prose Comparison: Reconciles operating system discrepancies by converting Windows carriage returns (CRLF) to Unix line feeds (LF) and stripping trailing whitespace at line ends. Crucially, the normalizer respects leading spaces, ensuring Python scripts, indented code blocks, and markdown list hierarchies remain immune from false matches.
- Frontmatter-Neutral Comparison: Strips YAML metadata blocks located between opening and closing
---fences. This allows you to identify notes that contain identical substantive prose even when their metadata properties (such as creation dates, author names, or category tags) differ.
Deterministic election of the canonical document
When duplicate groups are uncovered, deciding which file to preserve requires deliberate rules. The engine executes an automated canonical ranking strategy:
- Penalty Scoring: File paths containing indicators like
backup,copy,draft,archive,old, ortempreceive penalty points, disqualifying them from canonical status. - Directory Depth Minimization: Documents stored in primary, shallow directories (e.g.
handbook/policy.md) are preferred over copies buried in nested subfolders. - Deterministic Tiebreaking: Alphabetical sorting ensures reproducible results across multiple analysis runs.
The elected canonical document is locked as preserved and is never targeted by staging commands.
Handling zero-byte stubs and atomic filesystem operations
Knowledge bases occasionally contain empty placeholder notes created by unresolved wikilinks or quick capture shortcuts. Zero-byte files share identical content digests by definition. The engine flags empty notes explicitly, enabling you to prune dead stubs without confusing them with substantive essay duplicates.
Furthermore, when operating on large repositories, file hashes are stored in memory using sparse array indexes, avoiding memory bloat and preserving battery efficiency on laptops during extensive vault audits.
Reversible staging protocols: why automated disk deletion is omitted
Direct, programmatic deletion within your notes directory carries unacceptable risks of data destruction. External processes, scheduled build tools, or operating system symlinks might depend on specific file paths.
To ensure complete safety, this tool generates a standalone Bash migration script. Running this script moves duplicate files into a safe local holding directory (.archive/duplicate_notes/), allowing you to review changes in your terminal and maintain a complete rollback path.
Related offline text analysis utilities
Discover complementary private utilities for auditing and optimizing local text documents:
- Unused Attachment Reviewer: Cross-reference media inventories against markdown links to identify dangling images and PDFs.
- Attachment Path Auditor: Scrutinize image embeds for broken paths, hardcoded desktop folders, and external host URLs.
- Vault Health Inspector: Run a full health scan across backlinks, frontmatter tags, and directory structures.
- All Tools Directory: Discover private browser-based utilities for Markdown workflows.