Skip to content
IT Onboarding Wiki
Artificial Intelligence
Prompting Fundamentals
Incorrect Character Encoding

Incorrect Character Encoding

When using generative AI systems (e.g. ChatGPT, Claude, or comparable tools), texts may appear formally correct while containing technically problematic characters. These issues typically surface during further processing in word processors, LaTeX, Markdown, content management systems, or source code. Common problem cases include em dashes and non‑standard whitespace characters.

AI models do not generate text character by character, but token‑based. In doing so, they often rely on typographically refined Unicode characters that are visually appropriate but not always compatible with technical target systems. The result is text that looks correct at first glance, yet causes errors or unexpected behavior when copied, rendered, parsed, or further processed.

Em Dash instead of Hyphen

Instead of a standard hyphen (-, ASCII 45), AI systems frequently output an em dash (—, Unicode U+2014) or an en dash (–, Unicode U+2013). These characters are typically used:

  • for typographic emphasis
  • as thought dashes
  • as substitutes for colons or parentheses

Typical Symptoms

  • Issues when importing into LaTeX, Markdown, or CSV
  • Incorrect word breaking or search behavior
  • Unexpected line breaks
  • Malfunctions in systems that expect ASCII characters

Examples

AI output
The use of AI—especially in academic contexts—requires clear rules.
Technically robust
The use of AI - especially in academic contexts - requires clear rules.

Unusual or Invisible Whitespace

AI systems sometimes use Unicode whitespace characters that differ from the standard space (U+0020). These are often visually indistinguishable but can cause significant technical issues. Common variants include:

  • Non‑breaking space (U+00A0)
  • Thin space (U+2009)
  • Narrow no‑break space (U+202F)
  • Zero‑width space (U+200B)

Typical Symptoms

  • Lines do not wrap correctly
  • Copy & paste results in “stuck together” words
  • Problems with automated text processing
  • Errors in regular expressions or parsing
  • Deviations in character or word counts

Causes

The root cause is not a classic “encoding error”, but rather the typographic optimization of AI output:

  • AI systems favor typographically correct characters
  • Models are trained on editorial and publication‑grade texts
  • Unicode is used by default
  • The technical target context (e.g. LaTeX, Markdown, code) is often not implicit to the model

Practical Recommendations

During Prompting

Explicitly specify the target system
Prompt
Output the text using **pure ASCII** only.
Prompt
Do **not** use typographic dashes, only standard hyphens.
Explicitly define the format
Prompt
The text will be further processed in Markdown.
Prompt
The text will be used in LaTeX.

Post‑processing

  • Before further use, texts should be:

    • pasted as “plain text”
    • checked in an editor that reveals Unicode characters
    • normalized automatically if necessary (e.g. replacing — with -)

  • In academic work:

    • pay particular attention to abstracts, quotations, and headings
    • perform a formal character check before submission or publication
Last updated on
Prompt Examples