1 of 19

Set notation in regular expressions

TC39 November 2020

​

Markus Scherer & Mathias Bynens

​

https://github.com/mathiasbynens/proposal-regexp-set-notation

2 of 19

Properties, character classes, combinations

3 of 19

Character properties

[0-9٠-٩۰-۹߀-߉०-९০-৯੦-੯ ૦-૯୦-୯௦-௯౦-౯...０-９...]

→

\p{Decimal_Number}

4 of 19

Property plus

  • additions (union)
  • exceptions (subtraction, set difference)
  • “both this property & that property” (intersection)

5 of 19

ES regex: only union

One notion of “word/identifier letters”:

[\p{Letter}\p{Mark}\p{Number}_]

​

6 of 19

Hack: Intersection via grouping+lookahead

“word/identifier letters, Khmer script only”:

(?:(?=\p{Script=Khmer})[\p{Letter}\p{Mark}\p{Number}_])

​

7 of 19

Hack: Subtraction via grouping+lookahead

“Non-ASCII digits”:

(?:(?![0-9])[\p{Decimal_Number}])

8 of 19

Hack: Resolve & hardcode

“Non-ASCII digits”:

[٠-٩۰-۹߀-߉०-९০-৯੦-੯ ૦-૯୦-୯௦-௯౦-౯...０-９...]

9 of 19

ES regex character classes with

  • intersection
  • subtraction
  • nested character classes

10 of 19

Some syntax for intersection

instead of

(?:(?=\p{Script=Khmer})[\p{Letter}\p{Mark}\p{Number}])

we could allow writing

[\p{Script=Khmer}&&[\p{Letter}\p{Mark}\p{Number}]]

Actual ES syntax TBD

11 of 19

Subtraction & nested classes

instead of

(?:(?![0-9])[\p{Decimal_Number}])

we could allow writing

[\p{Decimal_Number}--[0-9]]

Actual ES syntax TBD

12 of 19

Union with nested class

General support for nested classes – consistent & predictable syntax:

[\p{Letter}\p{Mark}\p{Number}[_-]]

​

13 of 19

More examples

“breaking spaces”

emoji characters except for the ASCII ones

non-script-specific combining marks�

Looking for “first letter in each script” starting from:

[\p{White_Space}--\p{Line_Break=Glue}]

[\p{Emoji}--[#*0-9]]

[\p{Nonspacing_Mark}&&[\p{Script=Inherited}�\p{Script=Common}]]

[\P{NFC_Quick_Check=No}--\p{Script=Common}--�\p{Script=Inherited}--\p{Script=Unknown}]

Actual ES syntax TBD

14 of 19

Other regex engines

ICU, Java, Perl (experimental), Python regex module, .Net, XML Schema, Xerces, Ruby

  • union & nested classes – all
  • intersection – ICU, Java, Perl, Python regex, Xerces, Ruby
  • subtraction/difference – ICU, Perl, Python regex, .Net, XML Schema, Xerces
  • symmetric difference – Perl, Python regex

15 of 19

ICU, Java, Perl (experimental), Python regex module, .Net, XML Schema, Xerces, Ruby

  • union & nested classes – all
  • intersection – ICU, Java, Perl, Python regex, Xerces, Ruby
  • subtraction/difference – ICU, Perl, Python regex, .Net, XML Schema, Xerces
  • symmetric difference – Perl, Python regex

language/implementation

union

subtraction

intersection

nested classes

symmetric difference

✅

✅

✅

✅

❌

✅

🤷 *

✅

✅

❌

✅

✅

✅

✅

✅

✅

✅

❌

✅

❌

✅

✅

❌

✅

❌

✅

✅

✅

✅

❌

Python regex

✅

✅

✅

✅

✅

Ruby

✅

❌

✅

❌

❌

ECMAScript prior to this proposal

✅

❌

❌

❌

❌

ECMAScript with this proposal

✅

✅

✅

✅

❌

​

16 of 19

Benefits

Simplicity: Shorter, more intuitive expressions

Avoiding errors:

  • Hardcoded character classes are very difficult to assess for errors
  • They also fall out of date; e.g., \p{Emoji} changes each year
  • Avoid bad mistakes like assuming letter = [a-zA-Z]
  • Lookaheads are more difficult to use correctly (and are slower)

17 of 19

Request:

Approve in principle support for

  • intersection
  • subtraction
  • nested character classes
    • also for union

18 of 19

[\p{Letter}-\p{Lowercase}]

[A-ZÀ-ÖØ-ÞĀĂĄĆĈĊČĎĐĒ ĔĖĘĚĜĞĠĢĤĦĨĪĬĮİIJĴĶĹĻĽ ĿŁŃŅŇŊŌŎŐŒŔŖŘŚŜŞŠŢŤŦŨ ŪŬŮŰŲŴŶŸŹŻŽƁƂƄƆƇƉ-ƋƎ-Ƒ ƓƔƖ-ƘƜƝƟƠƢƤƦƧƩƬƮƯƱ-ƳƵ ƷƸƻƼǀ-DžLJLjNJNjǍǏǑǓǕǗǙǛǞǠ ǢǤǦǨǪǬǮDZDzǴǶ-ǸǺǼǾȀȂȄȆȈ ȊȌȎȐȒȔȖȘȚȜȞȠȢȤȦȨȪȬȮȰȲ ȺȻȽȾɁɃ-ɆɈɊɌɎʔʹ-ʿˆ-ˑˬˮ ͰͲʹͶͿΆΈ-ΊΌΎΏΑ-ΡΣ-ΫϏϒ-ϔ ϘϚϜϞϠϢϤϦϨϪϬϮϴϷϹϺϽ-ЯѠѢ ѤѦѨѪѬѮѰѲѴѶѸѺѼѾҀҊҌҎҐҒҔ ҖҘҚҜҞҠҢҤҦҨҪҬҮҰҲҴҶҸҺҼҾ ӀӁӃӅӇӉӋӍӐӒӔӖӘӚӜӞӠӢӤӦӨ ӪӬӮӰӲӴӶӸӺӼӾԀԂԄԆԈԊԌԎԐԒ ԔԖԘԚԜԞԠԢԤԦԨԪԬԮԱ-Ֆՙא-ת ׯ-ײؠ-يٮٯٱ-ۓەۥۦۮۯۺ-ۼۿܐ ܒ-ܯݍ-ޥޱߊ-ߪߴߵߺࠀ-ࠕࠚࠤࠨࡀ-ࡘ ࡠ-ࡪࢠ-ࢴࢶ-ࢽऄ-हऽॐक़-ॡॱ-ঀঅ-ঌ এঐও-নপ-রলশ-হঽৎড়ঢ়য়-ৡৰৱ ৼਅ-ਊਏਐਓ-ਨਪ-ਰਲਲ਼ਵਸ਼ਸਹਖ਼-ੜ ਫ਼ੲ-ੴઅ-ઍએ-ઑઓ-નપ-રલળવ-હ ઽૐૠૡૹଅ-ଌଏଐଓ-ନପ-ରଲଳଵ-ହ ଽଡ଼ଢ଼ୟ-ୡୱஃஅ-ஊஎ-ஐஒ-கஙசஜஞ டணதந-பம-ஹௐఅ-ఌఎ-ఐఒ-నప-హ ఽౘ-ౚౠౡಀಅ-ಌಎ-ಐಒ-ನಪ-ಳವ-ಹ ಽೞೠೡೱೲഅ-ഌഎ-ഐഒ-ഺഽൎൔ-ൖൟ-ൡ ൺ-ൿඅ-ඖක-නඳ-රලව-ෆก-ะาำ เ-ๆກຂຄຆ-ຊຌ-ຣລວ-ະາຳຽເ-ໄ ໆໜ-ໟༀཀ-ཇཉ-ཬྈ-ྌက-ဪဿၐ-ၕ ၚ-ၝၡၥၦၮ-ၰၵ-ႁႎႠ-ჅჇჍჼᄀ-ቈ ቊ-ቍቐ-ቖቘቚ-ቝበ-ኈኊ-ኍነ-ኰኲ-ኵ ኸ-ኾዀዂ-ዅወ-ዖዘ-ጐጒ-ጕጘ-ፚᎀ-ᎏ Ꭰ-Ᏽᐁ-ᙬᙯ-ᙿᚁ-ᚚᚠ-ᛪᛱ-ᛸᜀ-ᜌ ᜎ-ᜑᜠ-ᜱᝀ-ᝑᝠ-ᝬᝮ-ᝰក-ឳៗៜᠠ-ᡸ ᢀ-ᢄᢇ-ᢨᢪᢰ-ᣵᤀ-ᤞᥐ-ᥭᥰ-ᥴᦀ-ᦫ ᦰ-ᧉᨀ-ᨖᨠ-ᩔᪧᬅ-ᬳᭅ-ᭋᮃ-ᮠᮮᮯ ᮺ-ᯥᰀ-ᰣᱍ-ᱏᱚ-ᱽᲐ-ᲺᲽ-Ჿᳩ-ᳬ ᳮ-ᳳᳵᳶᳺḀḂḄḆḈḊḌḎḐḒḔḖḘḚḜ ḞḠḢḤḦḨḪḬḮḰḲḴḶḸḺḼḾṀṂṄṆ ṈṊṌṎṐṒṔṖṘṚṜṞṠṢṤṦṨṪṬṮṰ ṲṴṶṸṺṼṾẀẂẄẆẈẊẌẎẐẒẔẞẠẢ ẤẦẨẪẬẮẰẲẴẶẸẺẼẾỀỂỄỆỈỊỌ ỎỐỒỔỖỘỚỜỞỠỢỤỦỨỪỬỮỰỲỴỶ ỸỺỼỾἈ-ἏἘ-ἝἨ-ἯἸ-ἿὈ-ὍὙὛ ὝὟὨ-Ὧᾈ-ᾏᾘ-ᾟᾨ-ᾯᾸ-ᾼῈ-ῌῘ-Ί Ῠ-ῬῸ-ῼℂℇℋ-ℍℐ-ℒℕℙ-ℝℤΩℨ K-ℭℰ-ℳℵ-ℸℾℿⅅↃⰀ-ⰮⱠⱢ-ⱤⱧ ⱩⱫⱭ-ⱰⱲⱵⱾ-ⲀⲂⲄⲆⲈⲊⲌⲎⲐⲒⲔⲖ ⲘⲚⲜⲞⲠⲢⲤⲦⲨⲪⲬⲮⲰⲲⲴⲶⲸⲺⲼⲾⳀ ⳂⳄⳆⳈⳊⳌⳎⳐⳒⳔⳖⳘⳚⳜⳞⳠⳢⳫⳭⳲⴰ-ⵧ ⵯⶀ-ⶖⶠ-ⶦⶨ-ⶮⶰ-ⶶⶸ-ⶾⷀ-ⷆⷈ-ⷎ ⷐ-ⷖⷘ-ⷞⸯ々〆〱-〵〻〼ぁ-ゖゝ-ゟァ-ヺ ー-ヿㄅ-ㄯㄱ-ㆎㆠ-ㆺㇰ-ㇿ㐀-䶵一-鿯 ꀀ-ꒌꓐ-ꓽꔀ-ꘌꘐ-ꘟꘪꘫꙀꙂꙄꙆꙈꙊꙌ ꙎꙐꙒꙔꙖꙘꙚꙜꙞꙠꙢꙤꙦꙨꙪꙬꙮꙿꚀꚂꚄ ꚆꚈꚊꚌꚎꚐꚒꚔꚖꚘꚚꚠ-ꛥꜗ-ꜟꜢꜤꜦꜨ ꜪꜬꜮꜲꜴꜶꜸꜺꜼꜾꝀꝂꝄꝆꝈꝊꝌꝎꝐꝒꝔ ꝖꝘꝚꝜꝞꝠꝢꝤꝦꝨꝪꝬꝮꝹꝻꝽꝾꞀꞂꞄꞆ ꞈꞋꞍꞏꞐꞒꞖꞘꞚꞜꞞꞠꞢꞤꞦꞨꞪ-ꞮꞰ-Ꞵ ꞶꞸꞺꞼꞾꟂꟄ-Ᶎꟷꟻ-ꠁꠃ-ꠅꠇ-ꠊꠌ-ꠢ ꡀ-ꡳꢂ-ꢳꣲ-ꣷꣻꣽꣾꤊ-ꤥꤰ-ꥆꥠ-ꥼ ꦄ-ꦲꧏꧠ-ꧤꧦ-ꧯꧺ-ꧾꨀ-ꨨꩀ-ꩂꩄ-ꩋ ꩠ-ꩶꩺꩾ-ꪯꪱꪵꪶꪹ-ꪽꫀꫂꫛ-ꫝꫠ-ꫪ ꫲ-ꫴꬁ-ꬆꬉ-ꬎꬑ-ꬖꬠ-ꬦꬨ-ꬮꯀ-ꯢ 가-힣ힰ-ퟆퟋ-ퟻ豈-舘並-龎יִײַ-ﬨשׁ-זּ טּ-לּמּנּסּףּפּצּ-ﮱﯓ-ﴽﵐ-ﶏﶒ-ﷇﷰ-ﷻ ﹰ-ﹴﹶ-ﻼＡ-Ｚｦ-ﾾￂ-ￇￊ-ￏￒ-ￗ ￚ-ￜ𐀀-𐀋𐀍-𐀦𐀨-𐀺𐀼𐀽𐀿-𐁍𐁐-𐁝𐂀-𐃺 𐊀-𐊜𐊠-𐋐𐌀-𐌟𐌭-𐍀𐍂-𐍉𐍐-𐍵𐎀-𐎝 𐎠-𐏃𐏈-𐏏𐐀-𐐧𐑐-𐒝𐒰-𐓓𐔀-𐔧𐔰-𐕣 𐘀-𐜶𐝀-𐝕𐝠-𐝧𐠀-𐠅𐠈𐠊-𐠵𐠷𐠸𐠼𐠿-𐡕 𐡠-𐡶𐢀-𐢞𐣠-𐣲𐣴𐣵𐤀-𐤕𐤠-𐤹𐦀-𐦷𐦾 𐦿𐨀𐨐-𐨓𐨕-𐨗𐨙-𐨵𐩠-𐩼𐪀-𐪜𐫀-𐫇𐫉-𐫤 𐬀-𐬵𐭀-𐭕𐭠-𐭲𐮀-𐮑𐰀-𐱈𐲀-𐲲𐴀-𐴣 𐼀-𐼜𐼧𐼰-𐽅𐿠-𐿶𑀃-𑀷𑂃-𑂯𑃐-𑃨𑄃-𑄦 𑅄𑅐-𑅲𑅶𑆃-𑆲𑇁-𑇄𑇚𑇜𑈀-𑈑𑈓-𑈫𑊀-𑊆 𑊈𑊊-𑊍𑊏-𑊝𑊟-𑊨𑊰-𑋞𑌅-𑌌𑌏𑌐𑌓-𑌨 𑌪-𑌰𑌲𑌳𑌵-𑌹𑌽𑍐𑍝-𑍡𑐀-𑐴𑑇-𑑊𑑟𑒀-𑒯 𑓄𑓅𑓇𑖀-𑖮𑗘-𑗛𑘀-𑘯𑙄𑚀-𑚪𑚸𑜀-𑜚𑠀-𑠫 𑢠-𑢿𑣿𑦠-𑦧𑦪-𑧐𑧡𑧣𑨀𑨋-𑨲𑨺𑩐𑩜-𑪉 𑪝𑫀-𑫸𑰀-𑰈𑰊-𑰮𑱀𑱲-𑲏𑴀-𑴆𑴈𑴉𑴋-𑴰 𑵆𑵠-𑵥𑵧𑵨𑵪-𑶉𑶘𑻠-𑻲𒀀-𒎙𒒀-𒕃𓀀-𓐮 𔐀-𔙆𖠀-𖨸𖩀-𖩞𖫐-𖫭𖬀-𖬯𖭀-𖭃𖭣-𖭷 𖭽-𖮏𖹀-𖹟𖼀-𖽊𖽐𖾓-𖾟𖿠𖿡𖿣𗀀-𘟷𘠀-𘫲 𛀀-𛄞𛅐-𛅒𛅤-𛅧𛅰-𛋻𛰀-𛱪𛱰-𛱼𛲀-𛲈 𛲐-𛲙𝐀-𝐙𝐴-𝑍𝑨-𝒁𝒜𝒞𝒟𝒢𝒥𝒦𝒩-𝒬 𝒮-𝒵𝓐-𝓩𝔄𝔅𝔇-𝔊𝔍-𝔔𝔖-𝔜𝔸𝔹𝔻-𝔾 𝕀-𝕄𝕆𝕊-𝕐𝕬-𝖅𝖠-𝖹𝗔-𝗭𝘈-𝘡𝘼-𝙕 𝙰-𝚉𝚨-𝛀𝛢-𝛺𝜜-𝜴𝝖-𝝮𝞐-𝞨𝟊𞄀-𞄬 𞄷-𞄽𞅎𞋀-𞋫𞠀-𞣄𞤀-𞤡𞥋𞸀-𞸃𞸅-𞸟𞸡 𞸢𞸤𞸧𞸩-𞸲𞸴-𞸷𞸹𞸻𞹂𞹇𞹉𞹋𞹍-𞹏𞹑𞹒𞹔 𞹗𞹙𞹛𞹝𞹟𞹡𞹢𞹤𞹧-𞹪𞹬-𞹲𞹴-𞹷𞹹-𞹼𞹾 𞺀-𞺉𞺋-𞺛𞺡-𞺣𞺥-𞺩𞺫-𞺻𠀀-𪛖𪜀-𫜴 𫝀-𫠝𫠠-𬺡𬺰-𮯠丽-𪘀]

19 of 19

[\p{Letter}-\p{Lowercase}](-[\p{Letter}-\p{Lowercase}])*

Exercise for the reader...