Free tool · works now

Exact Duplicate Note Finder

Identify byte-identical and normalized duplicate Markdown notes, reclaim disk storage, and stage copies safely.

Runs 100% in your browser. Find byte-identical and normalized duplicate notes across your vault, preserve canonical originals, and generate safe staging scripts.

4Notes scanned
2Unique notes
1Duplicate groups
1Duplicate copies
342 BReclaimable bytes
Multi-note text input4 notes parsed
Identified Duplicate Groups1 groups
Group ID & DigestCanonical Note (Preserved)Duplicate Copies (Staged)File SizeReclaimableContent Excerpt
dup-group-1
342-2fe11c10
Canonical
notes/q3-strategy.md
Copybackups/q3-strategy-copy.md
342 B342 B
--- title: Q3 Product Strategy tags: [strategy, executive] updated: 2026-07-01 --- # Q3 Product Strategy & Execution Plan We focus on three primary pillars for our local-first sof
Safe Shell Staging CommandNon-destructive staging
#!/usr/bin/env bash
# Script generated by Exact Duplicate Note Finder
# Preserves primary canonical notes and stages duplicate copies
set -euo pipefail

# Total duplicate copies to stage: 1
mkdir -p .archive/duplicate_notes
mv "backups/q3-strategy-copy.md" .archive/duplicate_notes/
Spreadsheet Deduplication LedgerFormula protected
"Group ID","Status","File Path","Byte Size","Reclaimable Bytes","Comparison Mode"
"dup-group-1","Canonical (Kept)","notes/q3-strategy.md","342","0","exact"
"dup-group-1","Duplicate (Stage)","backups/q3-strategy-copy.md","342","342","exact"

Content hashing and collision detection across text archives

As personal knowledge graphs expand, redundant documents accumulate across disparate subdirectories. Multi-device background synchronizers (such as Syncthing, iCloud Drive, Nextcloud, and Git merges) frequently spawn duplicate markdown files following connection timeouts or merge conflicts. Similarly, manual project backups, duplicated templates, and clippers leave identical copies scattered across nested folders.

These redundant files degrade search precision, cause wikilinks to resolve unpredictably to stale copies, and increase repository size.

Hashing mechanics: Fowler-Noll-Vo digests and byte metrics

To evaluate hundreds of documents instantaneously without sending private records to an external cloud server, the engine computes deterministic 32-bit FNV-1a (Fowler-Noll-Vo) digests paired with exact byte-length checks. This multi-attribute fingerprinting maps incoming text files into distinct hash buckets with sub-millisecond execution times.

Comparison specifications: exact binary vs normalized semantics

Depending on your workflow, text duplication presents itself in different forms. This utility equips you with three tailored comparison profiles:

  • Exact Byte Comparison: Requires character-for-character binary identity. Every byte, line ending, and whitespace character must align perfectly. This is the strictest mode for detecting true binary duplicates.
  • Normalized Prose Comparison: Reconciles operating system discrepancies by converting Windows carriage returns (CRLF) to Unix line feeds (LF) and stripping trailing whitespace at line ends. Crucially, the normalizer respects leading spaces, ensuring Python scripts, indented code blocks, and markdown list hierarchies remain immune from false matches.
  • Frontmatter-Neutral Comparison: Strips YAML metadata blocks located between opening and closing --- fences. This allows you to identify notes that contain identical substantive prose even when their metadata properties (such as creation dates, author names, or category tags) differ.

Deterministic election of the canonical document

When duplicate groups are uncovered, deciding which file to preserve requires deliberate rules. The engine executes an automated canonical ranking strategy:

  1. Penalty Scoring: File paths containing indicators like backup, copy, draft, archive, old, or temp receive penalty points, disqualifying them from canonical status.
  2. Directory Depth Minimization: Documents stored in primary, shallow directories (e.g. handbook/policy.md) are preferred over copies buried in nested subfolders.
  3. Deterministic Tiebreaking: Alphabetical sorting ensures reproducible results across multiple analysis runs.

The elected canonical document is locked as preserved and is never targeted by staging commands.

Handling zero-byte stubs and atomic filesystem operations

Knowledge bases occasionally contain empty placeholder notes created by unresolved wikilinks or quick capture shortcuts. Zero-byte files share identical content digests by definition. The engine flags empty notes explicitly, enabling you to prune dead stubs without confusing them with substantive essay duplicates.

Furthermore, when operating on large repositories, file hashes are stored in memory using sparse array indexes, avoiding memory bloat and preserving battery efficiency on laptops during extensive vault audits.

Reversible staging protocols: why automated disk deletion is omitted

Direct, programmatic deletion within your notes directory carries unacceptable risks of data destruction. External processes, scheduled build tools, or operating system symlinks might depend on specific file paths.

To ensure complete safety, this tool generates a standalone Bash migration script. Running this script moves duplicate files into a safe local holding directory (.archive/duplicate_notes/), allowing you to review changes in your terminal and maintain a complete rollback path.

Related offline text analysis utilities

Discover complementary private utilities for auditing and optimizing local text documents:

Questions

How does the Exact Duplicate Note Finder compare notes?

The engine hashes note content locally using deterministic 32-bit FNV-1a digests combined with byte-length checks. It supports three distinct comparison modes: exact byte-for-byte matching, normalized prose matching (which reconciles CRLF/LF line breaks and trailing spaces), and frontmatter-neutral matching (which strips YAML metadata headers to locate notes with identical bodies).

Why does the tool generate an archive shell script instead of deleting duplicate files directly?

Automated disk deletion in local note repositories can accidentally purge recent edits, unsaved drafts, or files referenced by external scripts. Instead of deleting files, the tool generates a reversible Bash script that moves duplicate copies into an archive folder (.archive/duplicate_notes/) while preserving your primary canonical note.

How does the tool choose which file is the primary canonical note?

The engine applies an intelligent heuristic: it prioritizes clean directory paths over backup or draft folders (penalizing paths containing backup, copy, draft, archive, or old), then favors shallower directory depths, and finally sorts alphabetically. The primary note is preserved and never moved.

Are indented code blocks and markdown tables protected during normalization?

Yes. The normalization algorithm only trims trailing whitespace at the ends of lines and standardizes newline characters. It strictly preserves leading indentation, list structures, code fences, and tabular alignment, ensuring that whitespace-sensitive syntax is never falsely merged.

Does this tool transmit my notes or files over the internet?

No. The entire duplicate detection, hashing, and script generation process executes 100% locally in your web browser. Zero network requests or telemetry transmissions are made.

Keep your local Markdown repository lean and conflict-free

Enjoy native plain text files, instant fuzzy search, and private on-device notes in Oyma.

tools exact duplicate note finder downloadDownload for Mac

No account needed. Intel Mac build