8000 Fix sticky unicode RegExp when lastIndex is inside a surrogate pair by Kernel-Hunter · Pull Request #876 · dop251/goja · GitHub
[go: up one dir, main page]

Skip to content

Fix sticky unicode RegExp when lastIndex is inside a surrogate pair - #876

Open
Kernel-Hunter wants to merge 2 commits into
dop251:masterfrom
Kernel-Hunter:fix-sticky-unicode-split-pair
Open

Kernel-Hunter wants to merge 2 commits into
dop251:masterfrom
Kernel-Hunter:fix-sticky-unicode-split-pair

Conversation

@Kernel-Hunter
Copy link
Copy Markdown

With the u flag, a match that starts inside a surrogate pair is reported at the start of the pair (that's what TestRegexpUnicodeAdvanceStringIndex #3 checks), so the global version works:

var re = /./gu;
re.lastIndex = 1;
re.exec("😀"); // ["😀"], index 0, lastIndex 2

The sticky version of the same thing fails:

var re = /./yu;
re.lastIndex = 1;
re.exec("😀"); // null, lastIndex 0 (V8 and SpiderMonkey: ["😀"], index 0)

"😀b".replace(re, "X") and re.test() fail the same way. The sticky check compares the match position with lastIndex itself, but the match is reported at lastIndex - 1.

The fix makes the sticky check in execRegexp and in findAllSubmatchIndexUnicode use the start of the pair when lastIndex points at its second half. Nothing changes when lastIndex is on a code point boundary.

One thing to be aware of: a pattern that matches a lone trail surrogate, like /\uDE00/yu at lastIndex 1, now matches the pair just like /\uDE00/gu already does in goja (V8 returns null for both). I kept it consistent with the existing global behaviour rather than changing that too.

Added TestRegexpStickyUnicodeLastIndexInSurrogatePair, which fails on master and passes now. go test . and go vet pass; I didn't run the tc39 suite locally.

…ate pair

With the u flag, a match that starts in the middle of a surrogate pair is
reported at the start of the pair, so /./gu with lastIndex 1 on "\uD83D\uDE00"
finds the pair at index 0. The sticky check still compared the match
position with lastIndex itself, so /./yu failed on the same input (and
replace() and test() with it). It now compares with the start of the pair.
Comment thread regexp.go
savedRune, runes[mappedStart] = runes[mappedStart], second
defer func() { runes[mappedStart] = savedRune }()
// the match is reported at the start of the pair, so a sticky match must start there
start = posMap[mappedStart]
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With this change the sticky case now goes through the rune swap above, and that swap gives wrong answers when lastIndex is inside a pair:

var r = /\uDE00/yu; r.lastIndex = 1;
r.exec("\u{1F600}")   // this PR: match at 0, length 2, lastIndex 2. Node/V8: null
var r2 = /\u{1F600}/yu; r2.lastIndex = 1;
r2.exec("\u{1F600}")  // this PR and master: null. Node/V8: match at 0, lastIndex 2

., [^a] and (?:) look right only because they also match a lone trail surrogate. The g path on master has the same problem (/\uDE00/gu with lastIndex 1 matches the pair, /\u{1F600}/gu does not).

Per spec the lastIndex inside a pair maps to the code point that holds it, so the match should run from the pair start on the real code point. Dropping the swap (here and in findSubmatchIndexUnicode) and keeping your start = posMap[mappedStart] fixes all of these:

	if splitPair {
		// lastIndex is inside a surrogate pair: match from the start of the pair
		start = posMap[mappedStart]
	}

I tried that on top of this branch: the y and g cases above all match Node, go test ./... passes, and the test262 run (TestTC39) still passes. A test with /\uDE00/yu and /\u{1F600}/yu at lastIndex 1 would cover it.

@Kernel-Hunter
Copy link
Copy Markdown
Author

Thanks bardrop, you're right. I checked your examples against node and /\uDE00/yu and /\uDE00/gu with lastIndex 1 return null there, so the swap was wrong for the g path as well. I dropped it in both findSubmatchIndexUnicode and findAllSubmatchIndexUnicode and kept the start = posMap[mappedStart] mapping. I added a test covering your four cases (y and g, trail-only and full pair); it fails on the previous head and passes now, and go test ./... passes. TestTC39 is skipped in my checkout (no test262), so I haven't run that one myself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

0