lock table<\/code> statement, it will cause some problems when trying to integrate the PL\/SQL grammar.<\/p>\n\n\n\nHow do we find out whether a semicolon belongs to a PL\/SQL statement or to an SQL statement? Is this possible in the lexer? Well, I think it’s possible by doing some semantic predicate acrobatics, but I don’t think it’s sensible.<\/p>\n\n\n\n
Another approach is to use two lexers. The first one, extracting the relevant statement in the scope of the IslandSQL grammar. And the second lexer processes only the extracted statements. The parser uses the token stream from the second lexer. Perfect. However, we want to keep the original positions (line\/column) of the tokens in scope. They are important for navigating to the right place in the code. How do we do that?<\/p>\n\n\n\n
Keep Hidden Tokens as Whitespace<\/h2>\n\n\n\n
The idea is to replace all non-whitespace characters in hidden tokens with a space. This way the number of lines and the position in the line of all relevant tokens stay the same. The total number of characters is also the same (the number of bytes might change when multibyte characters are replaced).<\/p>\n\n\n\n
Here’s an example.<\/p>\n\n\n\n
6) Original SQL script<\/span>