Free tool · works now

Text Encoding Inspector

Diagnose unreadable mojibake characters, corrupted UTF-8 sequences, and legacy charsets in your notes.

Runs 100% in your browser. Inspect unreadable characters, detect Byte Order Marks (BOM), repair Windows-1252/UTF-8 mojibake, and identify irreversible replacement characters with honest confidence ratings.

Text Encoding Diagnostic Workbench

🔒 100% Browser-Local✨ Instant Mojibake Repair📏 BOM & Null Byte Audit🛡️ Formula-Guarded CSV
Diagnosed EncodingWINDOWS-1252Confidence: MEDIUM
15 mojibake artifacts restored

Detected Encoding Anomalies & Artifacts

PosIssue TypeArtifact SnippetDiagnosisSuggested Fix
114utf8-mojibake’Double-encoded UTF-8 artifact ("’" represents Right single quotation mark (U+2019)).Decodes cleanly as "’".
268utf8-mojibake‘Double-encoded UTF-8 artifact ("‘" represents Left single quotation mark (U+2018)).Decodes cleanly as "‘".
238utf8-mojibake“Double-encoded UTF-8 artifact ("“" represents Left double quotation mark (U+201C)).Decodes cleanly as "“".
114utf8-mojibakeâ€Double-encoded UTF-8 artifact ("â€" represents Right double quotation mark (U+201D)).Decodes cleanly as "”".
413utf8-mojibakeéDouble-encoded UTF-8 artifact ("é" represents Latin small letter e with acute (U+00E9)).Decodes cleanly as "é".
363utf8-mojibakeüDouble-encoded UTF-8 artifact ("ü" represents Latin small letter u with diaeresis (U+00FC)).Decodes cleanly as "ü".
467utf8-mojibake°Double-encoded UTF-8 artifact ("°" represents Degree sign (U+00B0)).Decodes cleanly as "°".
514utf8-mojibakeâ„¢Double-encoded UTF-8 artifact ("â„¢" represents Trade mark sign (U+2122)).Decodes cleanly as "™".
15Mojibake Patterns
0U+FFFD Replacements
0Control / Null Bytes
96%ASCII Character Ratio

The Character Encoding Diagnostics Guide: Understanding Mojibake, UTF-8, and Legacy Charsets

1. The Anatomy of Modern Text Encoding Failures

In contemporary computing, UTF-8 (Universal Character Set Transformation Format 8-bit) is the dominant character encoding standard across the web, modern operating systems, and developer toolchains. Designed by Ken Thompson and Rob Pike in 1992, UTF-8 is backward-compatible with 7-bit ASCII while capable of encoding all 1,114,112 Unicode code points using variable-width sequences of one to four 8-bit bytes.

Despite UTF-8’s ubiquity, digital note repositories frequently suffer from character corruption. Notes imported from legacy export dumps (such as Microsoft Windows applications, early WordPress blogs, older email clients, or European databases) were originally encoded in single-byte character sets such as Windows-1252 (CP-1252), ISO-8859-1 (Latin-1), or Mac Roman. When these single-byte streams are decoded as UTF-8—or conversely, when UTF-8 byte streams are decoded as Windows-1252—text transforms into garbled, unreadable character soup known globally as mojibake.

2. The Double-Encoding Mechanism: How UTF-8 Becomes Mojibake

Understanding mojibake requires tracing the exact byte transformations that occur during improper software transcoding. Consider the typographical right single quotation mark (’), code point U+2019:

  • UTF-8 Representation: In UTF-8, U+2019 is encoded as a three-byte sequence: 0xE2 0x80 0x99.
  • Windows-1252 Misinterpretation: If a text editor decodes those three bytes under Windows-1252 instead of UTF-8:
    • Byte 0xE2 maps to the character â (Latin small letter a with circumflex).
    • Byte 0x80 maps to the character € (Euro sign).
    • Byte 0x99 maps to the character ™ (Trade mark sign).
  • The Result: The single apostrophe is corrupted into the infamous three-character sequence: ’.

Similarly, the accented letter é (code point U+00E9, encoded in UTF-8 as two bytes 0xC3 0xA9) transforms into é when read as Latin-1. Because these multi-byte byte patterns are strictly defined by the Unicode specification, our inspection engine can invert the transformation deterministically, restoring original typographical symbols and accents without guessing.

3. Common Mojibake Artifacts and Their Unicode Equivalents

The table below illustrates the most frequent encoding corruption artifacts discovered across imported Markdown vaults:

Corrupted Mojibake SequenceIntended CharacterUnicode Code PointUTF-8 Byte Sequence
’Right single quote (’)U+20190xE2 0x80 0x99
“Left double quote (“)U+201C0xE2 0x80 0x9C
â€Right double quote (”)U+201D0xE2 0x80 0x9D
—Em dash (—)U+20140xE2 0x80 0x94
…Ellipsis (…)U+20260xE2 0x80 0xA6
é / èAccented e (é / è)U+00E9 / U+00E80xC3 0xA9 / 0xC3 0xA8
ü / ä / öGerman umlauts (ü / ä / ö)U+00FC / U+00E4 / U+00F60xC3 0xBC / 0xC3 0xA4 / 0xC3 0xB6
°Degree symbol (°)U+00B00xC2 0xB0

4. The Irreversibility of Unicode Replacement Characters (U+FFFD)

While mojibake represents reversible byte misinterpretation, the presence of the Unicode Replacement Character (U+FFFD, displayed as ) signals irreversible upstream data loss. When a conforming UTF-8 decoder encounters a byte sequence that violates UTF-8 syntax rules (for example, an invalid lead byte or an unexpected continuation byte), the decoder discards the invalid byte and inserts code point U+FFFD into the output string.

Because the original byte value was permanently discarded during that decoding step, no software can mathematically deduce what character originally existed in that position. Our inspector transparently catalogs every replacement character, counts occurrences, and flags the file's confidence level as Ambiguous. We explicitly reject presenting speculative guesses as verified truth.

5. Byte Order Marks (BOM) and Hidden Null Byte Hazards

A Byte Order Mark (BOM) is a sequence of bytes at the beginning of a text stream (0xEF 0xBB 0xBF for UTF-8; 0xFF 0xFE or 0xFE 0xFF for UTF-16) designed to signal endianness. In modern UTF-8 systems, a BOM is strongly discouraged by the Unicode Consortium because UTF-8 has no byte ordering variations.

When a UTF-8 file contains a BOM, static site generators, Markdown compilers, and configuration parsers (including YAML frontmatter processors) often fail to recognize the initial --- token because the hidden BOM characters precede it. Similarly, files copied from Windows UTF-16LE systems often contain interleaved null bytes (\x00) between every ASCII character. Our utility audits BOMs and null bytes, allowing you to clean your note files for universal cross-platform compatibility.

Frequently Asked Questions

Questions

What is mojibake and why does it occur in plain-text notes?

Mojibake (the Japanese term for "character transformation") occurs when text encoded in one character set is mistakenly decoded using another. The most common modern variant happens when a document containing UTF-8 multi-byte characters (such as smart quotes, em dashes, or accented vowels) is opened or converted by software expecting single-byte Windows-1252 or ISO-8859-1. A single two-byte or three-byte UTF-8 character becomes two or three garbled ASCII characters (such as ’ for right single quote).

Why does the tool report confidence as "ambiguous" for certain files?

Unlike file types with fixed structural signatures, plain text contains no mandatory header declaring its character set. Furthermore, when files contain Unicode Replacement Characters (�), the original byte sequence was already permanently discarded by an earlier application. In accordance with strict software engineering integrity, ambiguous encodings are transparently flagged as ambiguous rather than presenting guesses as certainty.

Can Unicode Replacement Characters (�) be automatically recovered?

No. The character U+FFFD (represented visually as a black question-mark diamond) indicates that an upstream decoder encountered bytes that violated the UTF-8 specification and irreversibly substituted the replacement symbol. Because the original byte values were discarded, true recovery requires re-exporting the note from its original source database or application.

What is a Byte Order Mark (BOM) and should it be removed?

A Byte Order Mark (0xFEFF) is a three-byte sequence placed at the beginning of a file to declare byte endianness in UTF-16 or UTF-32. When present in UTF-8 text files, it is unnecessary and often causes subtle compatibility bugs, such as breaking Jekyll or Astro YAML frontmatter parsers, causing terminal shell scripts to fail on line 1, or printing invisible leading blank spaces.

Are my notes or corrupted texts uploaded to cloud servers for analysis?

Never. All encoding pattern matching, byte audits, mojibake repairs, and candidate decoding calculations execute 100% locally within your browser client. No text ever leaves your workstation.

Keep Your Plain-Text Notes Pristine and Durable

Explore our browser-local tools for line ending normalization, unicode filename inspection, and safe clipboard cleaning.

tools text encoding inspector downloadDownload for Mac

No account needed. Intel Mac build