How a Single ß Can Break Everything

Vladimir Jovanović

Trusted expertise in

AI-powered communication

The trusted AI assistant for communication and productivity

Trusted by 40M+ people and 50K+ organizations

Works where you do

across 500,000+

web and desktop apps

Grammarly + Coda + Superhuman

Coda, the maker of powerful AI productivity tools, and Superhuman, the AI-native email app, are now a part of Grammarly.��Together, we aim to revolutionize how teams operate—enabling them to stay focused, aligned, and achieve results quickly.

Story time

• Username and profile picture
• Username is alphanumeric and limited to 30 characters
• There is a characters’ counter under the input field

Story time

• Username and profile picture
• Username is alphanumeric and limited to 30 characters
• There is a characters’ counter under the input field
• Profile picture is generated from the username

Changing requirements

• Allow users to add emojis to usernames
• if (newValue.length <= 30 && � newValue.all { it.isLetterOrDigit() || it.isWhitespace() }�)

​

Changing requirements

• Allow users to add emojis to usernames
• if (newValue.length <= 30 && � newValue.all { it.isLetterOrDigit() || it.isWhitespace() }�)

Changing requirements

• Allow users to add emojis to usernames
• if (newValue.length <= 30 && � newValue.all { it.isLetterOrDigit() || it.isWhitespace() }�)

Unicode

Unicode � ☐

Unicode � ☐

The Dark Ages Before Unicode

• Every System Had Its Own Encoding:
• ASCII: 128 characters (English only)
• ISO-8859-1: 256 characters (Western European)
• Shift-JIS: Japanese characters
• GB2312: Simplified Chinese
• KOI8-R: Cyrillic/Russian

The Dark Ages Before Unicode

• Every System Had Its Own Encoding:
• ASCII: 128 characters (English only)
• ISO-8859-1: 256 characters (Western European)
• Shift-JIS: Japanese characters
• GB2312: Simplified Chinese
• KOI8-R: Cyrillic/Russian

• 0xC4
• ISO-8859-1 -> Ä
• KOI8-R -> д

Unicode's Big Idea - Code Points

• Unicode's solution: assign every character a unique number called a code point.
• U+0041 = 'A'
• U+03A9 = 'Ω'
• U+4E2D = '中'
• U+1F680 = '🚀'
• The "U+" Notation is the standard way to write Unicode code points
• Always hexadecimal and always with U+ prefix

Unicode's Big Idea - Code Points

• Unicode's solution: assign every character a unique number called a code point.
• U+0041 = 'A'
• U+03A9 = 'Ω'
• U+4E2D = '中'
• U+1F680 = '🚀'
• The "U+" Notation is the standard way to write Unicode code points
• Always hexadecimal and always with U+ prefix
• Range: U+0000 to U+10FFFF (over 1 million possible characters)

From Code Points to Bytes

• Code point = The “idea” of a character
• UTF-8 and UTF-16 = Ways to encode that idea into bytes
• Like the number 42 written as: 42, 101010, XLII
• Two different strategies for encoding the same data

From Code Points to Bytes

• Code point = The “idea” of a character
• UTF-8 and UTF-16 = Ways to encode that idea into bytes
• Like the number 42 written as: 42, 101010, XLII
• Two different strategies for encoding the same data

• Unicode ≠ UTF-8
• Unicode defines the character set (e.g. U+0041 = 'A')
• UTF-8 and UTF-16 are encoding formats

UTF-8 - The Web's Choice

​

• UTF-8 Strategy: Use 1 to 4 bytes per character, with a clever self-describing system.
• Why UTF-8 Won the Internet:
• ASCII Compatibility: Characters U+0000 to U+007F use exactly 1 byte - identical to ASCII

UTF-8 - The Web's Choice

​

• UTF-8 Strategy: Use 1 to 4 bytes per character, with a clever self-describing system.
• Why UTF-8 Won the Internet:
• ASCII Compatibility: Characters U+0000 to U+007F use exactly 1 byte - identical to ASCII
• Self-Synchronizing: You can find character boundaries anywhere in the stream

UTF-8 - The Web's Choice

​

• UTF-8 Strategy: Use 1 to 4 bytes per character, with a clever self-describing system.
• Why UTF-8 Won the Internet:
• ASCII Compatibility: Characters U+0000 to U+007F use exactly 1 byte - identical to ASCII
• Self-Synchronizing: You can find character boundaries anywhere in the stream

• Usage Statistics:
• >99% of all websites use UTF-8
• Default for JSON, XML, HTML5
• Standard for most programming languages

UTF-16 - Android's Internal Choice

​

• All String objects use UTF-16 internally
• 2 or 4 bytes per character (surrogate pairs for some)
• "🚀".length() == 2, not 1

UTF-16 - Android's Internal Choice

​

• All String objects use UTF-16 internally
• 2 or 4 bytes per character (surrogate pairs for some)
• "🚀".length() == 2, not 1
• Why UTF-16? Java chose it in 1995 when Unicode was smaller

UTF-16 - Android's Internal Choice

​

• All String objects use UTF-16 internally
• 2 or 4 bytes per character (surrogate pairs for some)
• "🚀".length() == 2, not 1
• Why UTF-16? Java chose it in 1995 when Unicode was smaller

UTF-16 - Android's Internal Choice

​

• All String objects use UTF-16 internally
• 2 or 4 bytes per character (surrogate pairs for some)
• "🚀".length() == 2, not 1
• Why UTF-16? Java chose it in 1995 when Unicode was smaller

• The mismatch: Memory uses UTF-16, but I/O uses UTF-8
• Constant conversion happening behind the scenes

UTF-8 vs UTF-16 - A Character Comparison

​

UTF-8 vs UTF-16 - A Character Comparison

​

• Key Observations:
• UTF-8: 1-4 bytes, ASCII-compatible
• UTF-16: 2 or 4 bytes, consistent for most languages

Astral planes

​

Astral planes

​

Unicode Normalization

​

• NFC (Normalization Form Composed): precomposed characters where possible�Example: é = U+00E9
• NFD (Normalization Form Decomposed): base character plus combining mark�Example: e (U+0065) + ◌́ (U+0301) = é
• Both look the same, but code points are different

Unicode Normalization

​

• NFC (Normalization Form Composed): precomposed characters where possible�Example: é = U+00E9
• NFD (Normalization Form Decomposed): base character plus combining mark�Example: e (U+0065) + ◌́ (U+0301) = é
• Both look the same, but code points are different

• Sorting, searching, and matching may fail without normalization
• android.icu.text.Normalizer2

"🧑‍🧑‍🧒‍🧒".length == ?

"🧑‍🧑‍🧒‍🧒".length == 11

Why is "🧑‍🧑‍🧒‍🧒".length == 11?

🧑

🧑

🧒

🧒

=

🧑‍🧑‍🧒‍🧒

Why is "🧑‍🧑‍🧒‍🧒".length == 11?

Emoji Modifiers & Variation Selectors

​

Emoji Modifiers & Variation Selectors

​

Counting Grapheme Clusters

• Use import android.icu.text.BreakIterator
• Don’t use import java.text.BreakIterator

Counting Grapheme Clusters

class GetGraphemeClusterCount {

operator fun invoke(text: String): Int {

val boundary = BreakIterator.getCharacterInstance()

boundary.setText(text)

var count = 0

var start = boundary.first()

while (start != BreakIterator.DONE) {

val end = boundary.next()

if (end != BreakIterator.DONE) {

count++

}

start = end

}

return count

}

}

Be Careful With Regex

• Dot matches one UTF-16 code unit, not one grapheme cluster
• \w, \b are ASCII-centric, may fail with non-English text
• Can cause broken characters or validation errors

​

​

Using Regex Correctly

• Use \X
• Use Unicode properties: \p{L} (letters), \p{M} (marks), \p{N} (numbers)

​

​

Getting first grapheme cluster

private fun getFirstGraphemeCluster(text: String): String {

if (text.isEmpty()) return ""

val boundary = BreakIterator.getCharacterInstance()

boundary.setText(text)

val start = boundary.first()

val end = boundary.next()

return if (start != BreakIterator.DONE && end != BreakIterator.DONE) {

text.substring(start, end)

} else {

""

}

}

Back to the story

Back to the story

fun getInitials(username: String): String {

val words = username.splitIntoWords()

return when {

words.isEmpty() -> ""

words.size == 1 -> getFirstGraphemeCluster(words[0])

else -> getFirstGraphemeCluster(words[0]) +� getFirstGraphemeCluster(words[1])

}

}

Back to the story

fun getInitials(username: String): String {

val words = username.splitIntoWords()

return when {

words.isEmpty() -> ""

words.size == 1 -> getFirstGraphemeCluster(words[0])

else -> getFirstGraphemeCluster(words[0]) +� getFirstGraphemeCluster(words[1])

}.uppercase()

}

Things go bad 😳

require(getGraphemeClusterCount(initials) <= 2) {

"Profile image cannot display more than 2 grapheme clusters: '$initials'"

}

Profile image cannot display more than 2 grapheme clusters:

Things go bad 😳

require(getGraphemeClusterCount(initials) <= 2) {

"Profile image cannot display more than 2 grapheme clusters: '$initials'"

}

Profile image cannot display more than 2 grapheme clusters: SSH

Back to getInitials

fun getInitials(username: String): String {

val words = username.splitIntoWords()

return when {

words.isEmpty() -> ""

words.size == 1 -> getFirstGraphemeCluster(words[0])

else -> getFirstGraphemeCluster(words[0]) +� getFirstGraphemeCluster(words[1])

}.uppercase()

}

uppercase method

ßmart hacker

• 'ß'.uppercase() == "SS"
• ffi, ffl, ff, ʼn
• 'ẞ'.lowercase() == "ß"
• 'Σ'.lowercase() == "σ"
• "ςσ".uppercase() == "ΣΣ"
• 'ʃ'.lowercase() == "s"

ßmart hacker

• 'ß'.uppercase() == "SS"
• ffi, ffl, ff, ʼn
• 'ẞ'.lowercase() == "ß"
• 'Σ'.lowercase() == "σ"
• "ςσ".uppercase() == "ΣΣ"
• 'ʃ'.lowercase() == "s"

​

• Z̵͙͉̎a̵̩͛͛l̵͎̓g̶͚̺̊͠o̷̫̊
• 漢字, 𓂀, ,

​

Everyone Gets Tripped Up by Unicode

• iOS “Rainbow Flag 🏳️‍0🌈” crash: white flag + 0 + rainbow + variation selector caused iMessages to crash
• Android “Black Dot ⚫” bug: tapping a seemingly harmless black dot crashed WhatsApp and other apps due to thousands of hidden invisible Unicode characters

Everyone Gets Tripped Up by Unicode

• iOS “Rainbow Flag 🏳️‍0🌈” crash: white flag + 0 + rainbow + variation selector caused iMessages to crash
• Android “Black Dot ⚫” bug: tapping a seemingly harmless black dot crashed WhatsApp and other apps due to thousands of hidden invisible Unicode characters
• Unicode is complex, powerful, and worth mastering

T̸̛̙̞͙͔̪͕̝̀͐̀̍͠͝͝h̵̨͍̻̺̝̓̈́̏̋͊̔͋a̸̢̧̨̼̜̖̗̖̠͇͎͐̾͐͛͊̆̽͗̓̈͠ń̴̨̝͇̦̝̺͙̉̽k̸̹̣̯̠̖͔̺̻̱̘͎̍͗̓͊͋̀ ̶̛͓͕̻͈̺̍̋y̵̮̪͕̖̺̭̓͊̇̏̓̉͌͋ͅơ̵̢͖̗̟̪̯͚̤̞̗̈́̆͐̿͛͆͑́̓u̵̯̳̭͚͖͇͐̀͐̍̈́̆͛͒̑̑͋̔̕͠͠!̸̧̳̩̅͝

​

Vladimir Jovanović�https://vladimirj.dev� @vladimirj.dev‬

Questions�