Course contentsShow
← Data Science
Lesson 1135 of 2,14529. Data Wrangling WorkflowsFree lesson

Detecting and Fixing Encoding Issues

Identifying mojibake and encoding errors in datasets, and using tools to detect correct encodings.

Detecting and Fixing Encoding Issues

You'll learn how to identify garbled text (mojibake) in your datasets and use diagnostic tools to find and apply the correct encoding.

When you open a file expecting readable text but see symbols like é instead of é or ’ instead of an apostrophe, you're experiencing mojibake—garbled text caused by encoding mismatches. This happens when data encoded in one character set (like UTF-8 or Latin-1) is read using a different encoding assumption.

Recognizing the Problem

Common signs of encoding issues:

  • Strange accented characters where simple letters should be
  • Question marks or replacement characters (�)
  • Text that looks "doubled up" with extra symbols
  • Complete gibberish in place of readable words

These typically occur when reading CSV, text files, or data scraped from websites with non-ASCII characters.

Detection Strategy

Think of encoding detection like a translator guessing which language a document is written in. You can:

  1. Try common encodings explicitly: UTF-8 (most modern), Latin-1 (ISO-8859-1, Western European), or CP1252 (Windows)
  2. Use charset detection libraries that analyze byte patterns to suggest likely encodings
  3. Check source documentation or file metadata when available

Fixing the Issue

Once you identify the correct encoding, re-read the file specifying that encoding in your read_csv() or file-opening function using the encoding parameter you learned earlier. If the source encoding is truly unknown, detection tools provide probability scores to help you test candidates.

Key Takeaway: Encoding errors appear as garbled text (mojibake); fix them by identifying the correct character encoding through manual testing of common encodings or using detection libraries, then re-reading the file with the proper encoding specified.