Repository navigation
Remove $unsupported tokens in the regex lexers #269
Description
Currently we produce $unsupported for various things (e.g. lookahead, lookbehind in pcre) and then error about it.
My suggestion instead is that we do lex these correctly, and then error about them in the parser instead. This moves the concept of unsupportedness along a layer.
Eventually I'd like to also construct AST nodes for these, and only error about the unsupportedness when we come to do the AST->NFA conversion. This way we'd also have support for these features for e.g. AST -> regexp rendering (where FSM are not involved), but perhaps also opportunities to deal with them by AST rewriting.
Activity
sfstewman commentedon Oct 10, 2020
I think this makes a lot of sense.
For the PCRE dialect,
$unsupportedcurrently falls into four buckets:\b,\B,\K,\Z, and\G.(foo)\1can be transformed into(foo)(?:foo)which is compatible with DFAs and linear scanning.(*VERB)forms like(*COMMIT)and(*PRUNE). These are so specific to backtracking matchers (and PCRE) that I'm not sure if we want to include them. On the other hand, there aren't that many forms of this, so it may make sense.