Free tool · works now

Near-Duplicate Note Reviewer

Detect overlapping drafts and repeated material across Markdown notes with token-based Jaccard similarity.

Runs 100% in your browser. Surface overlapping drafts, repeated passages, and near-duplicate notes with transparent token-based Jaccard similarity.

4Notes analyzed
2Candidate pairs
1High overlap
1Short generic
56%Average score
Multi-note text collection4 notes parsed
Near-Duplicate Candidate Pairs & Overlap Evidence2 pairs
Pair DocumentsSimilarityConfidence ClassificationSide-by-Side Overlap EvidenceGuidance & Action
A:drafts/incident-postmortem-v1.md
B:reports/incident-postmortem-final.md
79%
61 shared bigrams
High Overlap
Excerpt A:The root cause was identified as an unindexed foreign key join in the user session lookup table.
Excerpt B:The root cause was identified as an unindexed foreign key join in the user session lookup table.
Substantial text overlap (79%). Candidate for consolidation or archiving secondary copy.
A:daily/2026-10-12-todo.md
B:daily/2026-10-13-todo.md
33%
4 shared bigrams
Short Generic
Excerpt A:# Daily Standup - Buy milk - Check server logs - Update tickets
Excerpt B:- Review pull requests
Short note detected (9 words). Similarity cannot guarantee redundancy. Manual inspection advised; no automatic deletion.
Reversible Staging InstructionsSafe non-destructive
#!/usr/bin/env bash
# Script generated by Near-Duplicate Note Reviewer
# Stages secondary copies from high-confidence overlapping notes
set -euo pipefail

# Total high-overlap secondary notes: 1
mkdir -p .archive/near_duplicate_notes
mv "reports/incident-postmortem-final.md" .archive/near_duplicate_notes/
Statistical Pair Audit LedgerSpreadsheet ready
"Pair ID","Note A","Note B","Similarity %","Confidence","Common Bigrams","Recommendation"
"pair-1","drafts/incident-postmortem-v1.md","reports/incident-postmortem-final.md","79%","High Overlap","61","Substantial text overlap (79%). Candidate for consolidation or archiving secondary copy."
"pair-2","daily/2026-10-12-todo.md","daily/2026-10-13-todo.md","33%","Short Generic Note","4","Short note detected (9 words). Similarity cannot guarantee redundancy. Manual inspection advised; no automatic deletion."

Statistical document similarity in decentralized knowledge repositories

Knowledge workers frequently iterate on documentation across separate files: drafting project postmortems, outlining client proposals, or duplicating standard operating procedures across team directories. Unlike byte-for-byte duplicates, these documents evolve independently. One version incorporates supplementary corrective actions, while another updates meeting attendee rolls or reorganizes paragraph sequences.

Standard filesystem deduplication tools rely strictly on hash equality and overlook these partial duplicates entirely. The Near-Duplicate Note Reviewer fills this gap by performing token-level statistical similarity audits directly in client-side memory.

Mathematical foundation: word-level bigram shingling and Jaccard metrics

To quantify textual overlap without requiring computationally prohibitive language models, the engine leverages set-theoretic shingling:

  1. Structural Tokenization: Frontmatter headers, code fences, and Markdown markup characters are stripped to expose clean lexical prose tokens.
  2. Bigram Shingle Generation: Adjacent word pairs (such as incident overview, latency exceeding, routing tier) are mapped into unique token sets for each document.
  3. Jaccard Coefficient Calculation: The intersection size of shared bigrams is divided by their total union size. This yields a transparent coefficient from 0% to 100% that accurately reflects shared prose patterns regardless of formatting differences.

The short-document paradox: guarding against false positive alerts

A frequent pitfall of automated text comparison algorithms is the short-document paradox. Brief notes—such as five-line daily standup logs or shopping checklists—contain a very limited vocabulary. When compared, two brief notes might mathematically exhibit a 60% similarity score simply because both list items like "Review tickets" and "Update team".

Recommending deletion for such notes would be disastrous. Our engine implements an explicit brevity guard: any document pair where either note contains fewer than 15 words is automatically classified as a Short Generic Note. The system warns the user that lexical similarity cannot guarantee factual redundancy and explicitly refrains from staging either file.

Transparent evidence extraction for human-in-the-loop review

Automated deduplication should never operate as an opaque black box. For every candidate pair discovered, the engine extracts the specific sentences exhibiting the highest concentration of shared bigram shingles. Presenting side-by-side excerpts enables you to verify the substance of the overlap in seconds before deciding whether to consolidate documents or retain both as distinct historical records.

Complementary private text engineering tools

Explore companion utilities designed for rigorous local knowledge base curation:

Questions

How does the Near-Duplicate Note Reviewer calculate text similarity?

The engine parses documents into normalized word sequences, constructs overlapping adjacent word bigrams (shingles), and calculates the mathematical Jaccard similarity coefficient between document pairs. This yields an exact percentage score representing the proportion of shared phrase patterns relative to total vocabulary.

Why are short generic notes protected from confident duplicate recommendations?

Brief files such as daily checklists or three-bullet meeting summaries naturally share common words (like buy, review, update, or meeting) purely by coincidence. The engine enforces a minimum word threshold; any pair involving a brief note is classified as Short Generic Note and explicitly warns against automated deletion.

How are evidence excerpts selected for comparison?

For every identified candidate pair, the algorithm scans both documents to extract the specific paragraphs containing the highest density of shared bigram tokens. This displays transparent side-by-side evidence of overlapping text directly in your browser.

What is the recommended workflow when two notes share 75% or more text?

High-overlap documents typically represent iterative versions of the same essay, meeting postmortem, or technical design document. We advise inspecting the side-by-side evidence, merging any unique insights into the primary canonical note, and using the generated shell script to stage the secondary draft in an archive folder.

Does this tool send my note contents to an external LLM or cloud API?

No. All tokenization, shingle set intersections, Jaccard mathematics, and evidence extractions occur entirely within your browser tab. Zero network transmissions or cloud AI requests are executed.

Detect overlapping drafts without cloud AI

Private on-device text intelligence and native Markdown editing in Oyma.

tools near duplicate note reviewer downloadDownload for Mac

No account needed. Intel Mac build