UTF-8 & Unicode: A Developer's Guide to Text That Doesn't Break
Top guides on UTF-8 stress the same points: declare encoding at every layer, use full UTF-8 in databases (utf8mb4), and test with international characters and emoji. This article ties those practices to APIs, JSON, and HTML so you avoid mojibake and double-encoding bugs.
Why UTF-8 is the default
UTF-8 is backward compatible with ASCII, uses 1–4 bytes per code point, and covers all of Unicode. Set <meta charset="UTF-8"> in HTML, send Content-Type: text/html; charset=UTF-8 (or application/json; charset=utf-8 for APIs), and save source files as UTF-8. Explicit beats implicit—proxies and CDNs vary by default.
Databases and utf8mb4
Use utf8mb4 in MySQL/MariaDB for full Unicode including 4-byte emoji. Pick collations that match your sorting and case rules. Application code should use multibyte-safe string handling; slicing by byte index corrupts UTF-8 sequences.
JSON, APIs, and escaping
JSON is Unicode text—fix encoding at the transport layer, not by over-escaping valid characters. Align with URL encoding vs HTML entities so you do not stack incompatible escapes.
NFC, NFD, and duplicate-looking strings
The same user-visible text can have multiple byte sequences: composed vs decomposed accents. Pick NFC or NFD at storage boundaries and apply it before unique indexes—otherwise “é” as one code point vs base+e accent collide in UX but not in your database. Search indexes may normalize aggressively; authentication identifiers should be byte-exact.
Grapheme clusters, emoji sequences, and ZWJ
Modern emoji combine skin tones, ZWJ joins, and flags from regional indicator pairs—naive character counts mis-size inputs and textarea limits. Test with diverse strings in fixtures; align truncation with grapheme boundaries when product policy requires “N characters” for bios or tweets.
Files, email headers, and punycode
Internationalized domain names surface as punycode in DNS; filenames on mixed OSes hit NFC normalization and reserved characters. Document how your upload pipeline preserves original names versus slugified storage keys, and cross-check with URL slug rules for public links.
Recommended tools
Inspect code points and escaped forms for debugging.
Percent-encode query strings without breaking UTF-8.
Escape markup for safe embedding in templates.
Pretty-print JSON payloads during encoding reviews.
Related guides
Frequently Asked Questions
What is mojibake?
Garbled text when UTF-8 bytes are read as a legacy encoding (or vice versa)—for example “Café” instead of “Café”.
Is UTF-8 the same as Unicode?
Unicode defines characters and code points; UTF-8 is an encoding that represents those code points as bytes on disk and on the wire.
Why use utf8mb4 in MySQL?
MySQL’s legacy utf8 type does not store full 4-byte UTF-8; utf8mb4 supports all Unicode including emoji.
Should I normalize Unicode strings?
For keys and deduplication, pick NFC or NFD consistently—otherwise visually identical strings can differ at the byte level.
Why does JavaScript string length differ from “character” count?
Strings are UTF-16 code units; emoji and some scripts use surrogate pairs—use segmenters or libraries for grapheme-aware length and slicing.
How do email addresses handle Unicode?
Internationalized email uses EAI standards; validate with IDNA/punycode for domains and follow RFC rules—never assume ASCII-only in global products.
Wrapping up
Treat UTF-8 as a system contract: browsers, APIs, databases, and logs must agree—or you will ship subtle corruption that only appears in production.
Unicode Converter →