Unicode Normalization and Equivalence
What you'll learn: How attackers exploit different Unicode representations of identical characters to slip past input validation filters.
The Core Problem
Unicode allows multiple byte sequences to represent the same visual character. For example, the letter "é" can be encoded two ways:
- Precomposed form (NFC): A single codepoint
U+00E9 - Decomposed form (NFD): Base letter "e" (
U+0065) + combining acute accent (U+0301)
These look identical to humans, but to a naive string comparison, they're completely different. An attacker can use this to bypass validation that checks for dangerous strings.
The Security Impact
Imagine your validation blocks filenames containing "../" to prevent directory traversal. An attacker submits a path using the decomposed form of special characters. Your filter doesn't recognize it—but the filesystem normalizes it later and accepts ../etc/passwd.
Similarly, consider username blacklists. If you block "admin" but the attacker registers "admın" (using Turkish dotless i, U+0131), they might bypass your check while the authentication system later treats them as equivalent.
Why This Matters
Different systems normalize Unicode at different stages:
- Your validation layer might check the raw input
- The database might normalize before storage
- The filesystem or browser might normalize before use
If normalization happens after validation, attackers slip through.
Example Attack Flow
- Validation checks for blacklisted string "script"
- Attacker sends "ſcript" (using long s,
U+017F) - Validation passes—strings don't match
- Later processing normalizes
ſ → s - Result: "script" executes despite being "blocked"
Key Takeaway: Always normalize Unicode input to a canonical form (like NFC) before validation, and ensure all downstream systems use the same normalization to prevent attackers from exploiting character equivalence.