Repository navigation
Fix sticky unicode RegExp when lastIndex is inside a surrogate pair - #876
Kernel-Hunter wants to merge 2 commits into
Conversation
…ate pair With the u flag, a match that starts in the middle of a surrogate pair is reported at the start of the pair, so /./gu with lastIndex 1 on "\uD83D\uDE00" finds the pair at index 0. The sticky check still compared the match position with lastIndex itself, so /./yu failed on the same input (and replace() and test() with it). It now compares with the start of the pair.
| savedRune, runes[mappedStart] = runes[mappedStart], second | ||
| defer func() { runes[mappedStart] = savedRune }() | ||
| // the match is reported at the start of the pair, so a sticky match must start there | ||
| start = posMap[mappedStart] |
There was a problem hiding this comment.
With this change the sticky case now goes through the rune swap above, and that swap gives wrong answers when lastIndex is inside a pair:
var r = /\uDE00/yu; r.lastIndex = 1;
r.exec("\u{1F600}") // this PR: match at 0, length 2, lastIndex 2. Node/V8: null
var r2 = /\u{1F600}/yu; r2.lastIndex = 1;
r2.exec("\u{1F600}") // this PR and master: null. Node/V8: match at 0, lastIndex 2
., [^a] and (?:) look right only because they also match a lone trail surrogate. The g path on master has the same problem (/\uDE00/gu with lastIndex 1 matches the pair, /\u{1F600}/gu does not).
Per spec the lastIndex inside a pair maps to the code point that holds it, so the match should run from the pair start on the real code point. Dropping the swap (here and in findSubmatchIndexUnicode) and keeping your start = posMap[mappedStart] fixes all of these:
if splitPair {
// lastIndex is inside a surrogate pair: match from the start of the pair
start = posMap[mappedStart]
}
I tried that on top of this branch: the y and g cases above all match Node, go test ./... passes, and the test262 run (TestTC39) still passes. A test with /\uDE00/yu and /\u{1F600}/yu at lastIndex 1 would cover it.
|
Thanks bardrop, you're right. I checked your examples against node and /\uDE00/yu and /\uDE00/gu with lastIndex 1 return null there, so the swap was wrong for the g path as well. I dropped it in both findSubmatchIndexUnicode and findAllSubmatchIndexUnicode and kept the start = posMap[mappedStart] mapping. I added a test covering your four cases (y and g, trail-only and full pair); it fails on the previous head and passes now, and go test ./... passes. TestTC39 is skipped in my checkout (no test262), so I haven't run that one myself. |
With the
uflag, a match that starts inside a surrogate pair is reported at the start of the pair (that's whatTestRegexpUnicodeAdvanceStringIndex#3 checks), so the global version works:The sticky version of the same thing fails:
"😀b".replace(re, "X")andre.test()fail the same way. The sticky check compares the match position withlastIndexitself, but the match is reported atlastIndex - 1.The fix makes the sticky check in
execRegexpand infindAllSubmatchIndexUnicodeuse the start of the pair whenlastIndexpoints at its second half. Nothing changes whenlastIndexis on a code point boundary.One thing to be aware of: a pattern that matches a lone trail surrogate, like
/\uDE00/yuatlastIndex1, now matches the pair just like/\uDE00/gualready does in goja (V8 returns null for both). I kept it consistent with the existing global behaviour rather than changing that too.Added
TestRegexpStickyUnicodeLastIndexInSurrogatePair, which fails on master and passes now.go test .andgo vetpass; I didn't run the tc39 suite locally.