Find and Remove Hidden Characters in AI Text or Any Pasted Text

The hidden characters that turn up in pasted text, AI output included, fall into two groups. Some are spaces that look normal but are not U+0020: the no-break space U+00A0 and the narrow no-break space U+202F. The others have no width at all: the zero width space U+200B, the joiners U+200C and U+200D, the direction marks U+200E and U+200F, the word joiner U+2060, the soft hyphen U+00AD and the byte order mark U+FEFF. To find them, look for any character in Unicode’s format category (Cf) or space category (Zs) that is not an ordinary space. In JavaScript that is the regular expression /(?! )[\p{Cf}\p{Zs}]/gu, and the one-liner below prints each match with its position and code point.

The ten characters, by their Unicode names

Each name below is the official character name from Unicode’s own data files (NamesList.txt, Unicode 18.0.0, read on 2026-10-10). The middle column says what the character is for. The last says what to do with it when you are cleaning ordinary prose.

Two of these are spaces you should replace, not delete. If you delete a no-break space, Total: 420 km becomes Total:420km. The other eight are invisible, so deleting them is usually safe. The exceptions are U+200D inside emoji, and U+200C and the direction marks in scripts that need them.

You can see where U+202F comes from without any AI involved. Node’s built-in number formatting in French puts it between the groups of digits:

const n = new Intl.NumberFormat('fr-FR').format(1234567.5);
console.log(JSON.stringify(n), [...n].filter((c) => c > '~')
  .map((c) => 'U+' + c.codePointAt(0).toString(16).toUpperCase()));

"1 234 567,5" [ 'U+202F', 'U+202F' ]

Those two “spaces” are not spaces to a split(' ') or a strict comparison. Copy a figure like that out of a web page or a document and you carry them with it.

A sample string to test with

Every snippet below runs on this one string. It holds all ten characters from the table, with the zero width joiner twice, inside a family emoji:

const text = 'Total:\u00A0420\u202Fkm. Co\u00ADoperate\u200B now,'
  + ' Auf\u200Clage \u{1F468}\u200D\u{1F469}\u200D\u{1F467}'
  + ' \u200Eleft \u200Fright e\u2060mail\uFEFF';

The same string in Python:

text = ('Total:\u00a0420\u202fkm. Co\u00adoperate\u200b now,'
        ' Auf\u200clage \U0001F468\u200d\U0001F469\u200d\U0001F467'
        ' \u200eleft \u200fright e\u2060mail\ufeff')

On screen it reads Total: 420 km. Cooperate now, Auflage 👨‍👩‍👧 left right email. Nothing looks wrong.

Find them with a JavaScript one-liner

Paste this into the browser’s developer console (F12, then the Console tab), or run it in Node. To check your own text, replace the sample with your text inside the quotes:

console.log([...text.matchAll(/(?! )[\p{Cf}\p{Zs}]/gu)]
  .map((m) => m.index + ' U+' + m[0].codePointAt(0)
    .toString(16).toUpperCase().padStart(4, '0'))
  .join('\n'));

6 U+00A0
10 U+202F
17 U+00AD
25 U+200B
35 U+200C
43 U+200D
46 U+200D
50 U+200E
56 U+200F
64 U+2060
69 U+FEFF

Eleven hits: all ten characters, with U+200D twice. \p{Cf} is Unicode’s “format” category, which holds every zero-width character in the table. \p{Zs} is the space separators, which hold U+00A0, U+202F and the ordinary space, and (?! ) drops the ordinary space. The numbers on the left are positions in UTF-16 code units, the same positions text.slice() uses.

What it misses: characters that look blank but belong to other categories, such as U+3164 HANGUL FILLER (a letter, Lo) and U+2800 BRAILLE PATTERN BLANK (a symbol, So). It also says nothing about letters that only look like Latin ones, such as Cyrillic а in place of a.

Find them with a Python one-liner

Run this after the sample above. For a file, use text = open('pasted.txt', encoding='utf-8').read() instead. Python’s unicodedata module names each character for you:

import unicodedata as ud; print('\n'.join(f'{i} U+{ord(c):04X} {ud.name(c, "?")}' for i, c in enumerate(text) if c != ' ' and ud.category(c) in ('Cf', 'Zs')))

6 U+00A0 NO-BREAK SPACE
10 U+202F NARROW NO-BREAK SPACE
17 U+00AD SOFT HYPHEN
25 U+200B ZERO WIDTH SPACE
35 U+200C ZERO WIDTH NON-JOINER
42 U+200D ZERO WIDTH JOINER
44 U+200D ZERO WIDTH JOINER
47 U+200E LEFT-TO-RIGHT MARK
53 U+200F RIGHT-TO-LEFT MARK
61 U+2060 WORD JOINER
66 U+FEFF ZERO WIDTH NO-BREAK SPACE

The same eleven characters, but from the joiners on the positions are not the same. The two joiners sit at 43 and 46 in JavaScript and at 42 and 44 in Python. Python counts code points. JavaScript counts UTF-16 units, and each emoji in the family takes two of them. Both are right for their own language. Just do not take a position from one and use it in the other.

See them in VS Code

These setting names are from VS Code’s release notes for versions 1.63 and 1.64. The defaults are from VS Code’s own source, read on 2026-10-10.

• editor.unicodeHighlight.invisibleCharacters is on by default. Its description is “Controls whether characters that just reserve space or have no width at all are highlighted.”
• editor.unicodeHighlight.nonBasicASCII is off except in untrusted workspaces. Its description is “Controls whether all non-basic ASCII characters are highlighted. Only characters between U+0020 and U+007E, tab, line-feed and carriage-return are considered basic ASCII.” Every character in the table is outside that range.
• Markdown is the exception. The 1.63 notes say “Markdown documents are not highlighted by default”, and the built-in Markdown extension sets editor.unicodeHighlight.invisibleCharacters to false for Markdown.

So to inspect a paste, turn on the setting whose description covers everything, in settings.json:

{
  "editor.unicodeHighlight.nonBasicASCII": true,
  "[markdown]": {
    "editor.unicodeHighlight.nonBasicASCII": true
  }
}

That setting highlights every character outside basic ASCII. Accented letters, curly quotes and emoji get highlighted too. Turn it back off when you are done, or list characters you never want flagged in editor.unicodeHighlight.allowedCharacters.

See them in Notepad++

The Notepad++ user manual (npp-user-manual.org, read on 2026-10-10) documents this under View > Show Symbol:

• Show Non-Printing Characters draws each one in a small box, as its code point or its abbreviation. Which one you get is set in Settings > Preferences > Editing > Non-Printing Characters. The manual’s table for this option lists U+00A0, U+200B, U+200C, U+200D, U+200E, U+200F, U+202F, U+2060 and U+FEFF from this post’s table, among others. It does not list U+00AD SOFT HYPHEN, so the manual makes no promise about that one.
• Show All Characters switches on every Show Symbol option at once, including spaces, tabs and line endings.

To find all ten, including the soft hyphen, use Search > Find with Search Mode set to Regular expression. The manual’s \x{ℕℕℕℕ} syntax matches a character by its code point and works in UTF-8 and UTF-16 documents:

[\x{00A0}\x{202F}\x{00AD}\x{200B}\x{200C}\x{200D}\x{200E}\x{200F}\x{2060}\x{FEFF}]

To clean up in Notepad++, do it in two passes of Search > Replace, still in Regular expression mode. First replace [\x{00A0}\x{202F}] with a single space. Then replace [\x{00AD}\x{200B}\x{200C}\x{200E}\x{200F}\x{2060}\x{FEFF}] with nothing. U+200D is left out of the second pass on purpose (see below).

Remove them without breaking emoji

The tempting fix is to delete everything in \p{Cf}. That takes the zero width joiners out of emoji. A family emoji is three people joined by two U+200D characters. Here is what deleting them does, counted with Intl.Segmenter, the browser and Node API that splits text into the characters a reader sees:

const seg = new Intl.Segmenter('en', { granularity: 'grapheme' });
const count = (s) => [...seg.segment(s)].length;
const family = '\u{1F468}\u200D\u{1F469}\u200D\u{1F467}';
const naive = family.replace(/\u200D/g, '');
console.log('as typed:', family, count(family), 'grapheme(s)');
console.log('naive:   ', naive, count(naive), 'grapheme(s)');

as typed: 👨‍👩‍👧 1 grapheme(s)
naive:    👨👩👧 3 grapheme(s)

One family becomes 1 grapheme → 3 graphemes: a man, a woman and a girl standing next to each other. The same happens to every emoji built with U+200D, including professions, skin tone families and the rainbow flag.

So clean in three steps. Replace the two spaces with an ordinary space. Delete the pure invisibles. Delete U+200D only where the next character is not an emoji:

const clean = (s) => s
  .replace(/[\u00A0\u202F]/g, ' ')
  .replace(/[\u00AD\u200B\u200C\u200E\u200F\u2060\uFEFF]/g, '')
  .replace(/\u200D(?!\p{Extended_Pictographic})/gu, '');
console.log(clean(text));
console.log('family after clean():', count(clean(family)), 'grapheme(s)');

Total: 420 km. Cooperate now, Auflage 👨‍👩‍👧 left right email
family after clean(): 1 grapheme(s)

The family stays one grapheme. Run the one-liner from the top again on the cleaned text and the only hits left are the two joiners inside the emoji.

The same clean-up in Python. The standard library has no Extended_Pictographic class, so this version keeps a joiner when the next character is in the main emoji ranges. That covers the family, but it is a looser test than the JavaScript one:

import re
def clean(s):
    s = re.sub(r'[\u00a0\u202f]', ' ', s)
    s = re.sub(r'[\u00ad\u200b\u200c\u200e\u200f\u2060\ufeff]', '', s)
    return re.sub(r'\u200d(?![\U0001F000-\U0001FAFF\u2600-\u27bf])', '', s)
print(clean(text))

Total: 420 km. Cooperate now, Auflage 👨‍👩‍👧 left right email

Where this clean-up is wrong: text in Persian, Arabic, Hebrew or Indic scripts. There U+200C and U+200D change how letters join, and the direction marks keep mixed-direction text in order, so do not run it on that text. Keep the find step and decide one character at a time.

Are these characters a watermark?

You will read claims online that hidden characters in AI text are a watermark. For this post I tried to open OpenAI’s own page on text provenance, and the server refused the request (HTTP 403). I found no primary source I could open that says any assistant adds these characters on purpose. So this post makes no claim either way. The steps above show you what is in your text, whatever put it there.

To see every code point in a paste named, without running any code, paste it into polyatic’s Unicode inspector.

← Back to all posts

Follow new posts by RSS: paste dankdev.com/feed.xml into any feed reader. It is a plain XML file — no signup, no email address, no account. Posts land there the same day they go up, and there is nothing to unsubscribe from later.

Back to the top