In the last episode, we extended the IslandSQL grammar to cover all DML statements as a single lexer token. Now it’s time to handle the complete grammar for one DML statement. The simplest one is lock table. A good reason to start with it and lay the foundation for the other DML commands.

The full source code is available on GitHub and the binaries on Maven Central.

Grammars in SQL Scripts

When using SQL scripts, we work with several grammars. There is always more than one grammar involved. It depends on your use case how much more.

The candidates when working with an Oracle Database 21c are:

• SQL*Plus
• SQLcl
• PGQL
• SQL
• PL/SQL
• Java
• more hidden in strings and LOBs such as XML, XSLT, JSON, …

The number of grammars is growing. For example, we expect JavaScript stored procedures in Oracle Database 23c.

In the previous episodes, we have primarily dealt with SQL*Plus and SQL as a whole. Before we deal with a specific SQL statement such as lock table, we need to know where a SQL statement starts and where it ends. The start seems obvious, but the end? Is the fragment SQL_END correctly describing the end of a SQL statement?

fragment SQL_END:
      EOF
    | (';' [ \t]* SINGLE_NL?)
    | SLASH_END
;

Where Does a SQL Statement End?

A common misconception is, that a SQL statement ends with a semicolon. This seems to be true when you only look at the syntax per statement in the Oracle Database documentation, but it is not. Here’s an example:

begin
   execute immediate '
      lock table dept in exclusive mode
   ';
end;
/

The anonymous PL/SQL block executes a dynamic lock table statement on line 3. Please note that the lock table statement starts with whitespace and ends with whitespace. We do not pass a semicolon as part of the execute immediate statement. This anonymous PL/SQL block completes successfully when the connected user can lock emp.

However, when we add a semicolon at the end of the lock table statement we get the following error:

Error starting at line : 1 in command -
begin
   execute immediate '
      lock table dept in exclusive mode;
   ';
end;
Error report -
ORA-00933: SQL command not properly ended
ORA-06512: at line 2
00933. 00000 -  "SQL command not properly ended"
*Cause:    
*Action:

It’s not allowed to terminate a SQL statement with a semicolon in dynamic SQL and therefore also when executing SQL via JDBC or ODBC.

But what is the semicolon for? Well, it terminates a SQL statement within a SQL*Plus or SQLcl script. In SQL*Plus you can change the behaviour using the set sqlterminator command.

The following script works in SQL*Plus (but not in SQLcl, it’s a documented limitation):

set sqlterminator $
lock table dept in exclusive mode$

The semicolon is the default terminator of SQL statements in SQL scripts. The semicolon is part of the SQL*Plus grammar but not part of the SQL grammar. However, it is more. It is also the only supported statement terminator in PL/SQL as the following SQL*Plus script shows.

set sqlterminator $
begin
   lock table dept in exclusive mode;
end;
/

The semicolon on line 4 terminates the anonymous PL/SQL block. It’s part of the PL/SQL grammar. The final slash is not part of the anonymous PL/SQL block. It is an alternative to the SQL*Plus run command. It sends the buffer (the anonymous PL/SQL block) to the database server.

By the way, the Oracle Database documentation explains why it uses a semicolon in its grammar. Here is the corresponding quote:

Note: SQL statements are terminated differently in different programming environments. This documentation set uses the default SQL*Plus character, the semicolon (;).
— Lexical Conventions, SQL Language Reference. Oracle Datase 21c

Yes, the semicolon is part of the SQL*Plus grammar. There is no common sequence of characters for identifying the end of an SQL statement.

What Are We Going to Do Now?

I initially wanted to use ANTLR modes to handle the complete grammar of chosen statements. However, ANTLR modes require that you can identify the start and the end of a mode. For the lock mode statement, the start is the lock keyword and the end is the SQL_END fragment as we used before. We could also use just the semicolon to determine the end. While this works for the lock table statement, it will cause some problems when trying to integrate the PL/SQL grammar.

How do we find out whether a semicolon belongs to a PL/SQL statement or to an SQL statement? Is this possible in the lexer? Well, I think it’s possible by doing some semantic predicate acrobatics, but I don’t think it’s sensible.

Another approach is to use two lexers. The first one, extracting the relevant statement in the scope of the IslandSQL grammar. And the second lexer processes only the extracted statements. The parser uses the token stream from the second lexer. Perfect. However, we want to keep the original positions (line/column) of the tokens in scope. They are important for navigating to the right place in the code. How do we do that?

Keep Hidden Tokens as Whitespace

The idea is to replace all non-whitespace characters in hidden tokens with a space. This way the number of lines and the position in the line of all relevant tokens stay the same. The total number of characters is also the same (the number of bytes might change when multibyte characters are replaced).

Here’s an example.

/* ===========================================
 * ignore multiline comment
 * =========================================== */
select * from dual;

rem ignore remark: select * from dual;

-- ignore single line comment
lock table dept in exclusive mode;

After the transformation, the script should look like this.

..............................................
...........................
.................................................
select * from dual;

......................................

.............................
lock table dept in exclusive mode;

Please note that a dot(.) represents a replaced character. Read a dot as a space.

We can use this converted script as input for the second lexer.

The implementation is relatively easy. I renamed the original lexer to IslandSqlScopeLexer und used this code for the transformation:

static public String getScopeText(CommonTokenStream tokenStream) {
    TokenStreamRewriter rewriter = new TokenStreamRewriter(tokenStream);
    tokenStream.fill();
    tokenStream.getTokens().stream()
            .filter(token -> token.getChannel() == Token.HIDDEN_CHANNEL
                    && token.getType() != IslandSqlScopeLexer.WS)
            .forEach(token -> {
                        StringBuilder sb = new StringBuilder();
                        token.getText().codePoints().mapToObj(c -> (char) c)
                                .forEach(c -> sb.append(c == '\t' || c == '\r' || c == '\n' ? c : ' '));
                        rewriter.replace(token, sb.toString());
                    }
            );
    return rewriter.getText();
}

The method gets a token stream as input and returns the transformed text (SQL script).

The ANTLR runtime comes with a TokenStreamRewriter that helps adding, deleting or changing tokens. We are only changing hidden tokens that are not of type whitespace. Tabs, carriage returns and line feeds are kept. Other characters are replaced by a space.

The New Lexer

After the preprocessing of the original input, we can concentrate on the islands. The sea is represented as a whitespace. This simplifies the logic of the lexer.

lexer grammar IslandSqlLexer;

options {
    superClass=IslandSqlLexerBase;
    caseInsensitive = true;
}

/*----------------------------------------------------------------------------*/
// Fragments to name expressions and reduce code duplication
/*----------------------------------------------------------------------------*/

fragment SINGLE_NL: '\r'? '\n';
fragment COMMENT_OR_WS: ML_COMMENT|SL_COMMENT|WS;
fragment SQL_TEXT: (ML_COMMENT|SL_COMMENT|STRING|.);
fragment SLASH_END: SINGLE_NL WS* '/' [ \t]* (EOF|SINGLE_NL);
fragment PLSQL_DECLARATION_END: ';'? [ \t]* (EOF|SLASH_END);
fragment SQL_END:
      EOF
    | (';' [ \t]* SINGLE_NL?)
    | SLASH_END
;

/*----------------------------------------------------------------------------*/
// Hidden tokens
/*----------------------------------------------------------------------------*/

WS: [ \t\r\n]+ -> channel(HIDDEN);
ML_COMMENT: '/*' .*? '*/' -> channel(HIDDEN);
SL_COMMENT: '--' .*? (EOF|SINGLE_NL) -> channel(HIDDEN);
CONDITIONAL_COMPILATION_DIRECTIVE: '$if' .*? '$end' -> channel(HIDDEN);

/*----------------------------------------------------------------------------*/
// Keywords
/*----------------------------------------------------------------------------*/

K_EXCLUSIVE: 'exclusive';
K_FOR: 'for';
K_IN: 'in';
K_LOCK: 'lock';
K_MODE: 'mode';
K_NOWAIT: 'nowait';
K_PARTITION: 'partition';
K_ROW: 'row';
K_SHARE: 'share';
K_SUBPARTITION: 'subpartition';
K_TABLE: 'table';
K_UPDATE: 'update';
K_WAIT: 'wait';

/*----------------------------------------------------------------------------*/
// Special characters
/*----------------------------------------------------------------------------*/

AT_SIGN: '@';
CLOSE_PAREN: ')';
COMMA: ',';
DOT: '.';
OPEN_PAREN: '(';
SEMI: ';';
SLASH: '/';

/*----------------------------------------------------------------------------*/
// Data types
/*----------------------------------------------------------------------------*/

STRING:
    'n'?
    (
          (['] .*? ['])+
        | ('q' ['] '[' .*? ']' ['])
        | ('q' ['] '(' .*? ')' ['])
        | ('q' ['] '{' .*? '}' ['])
        | ('q' ['] '<' .*? '>' ['])
        | ('q' ['] . {saveQuoteDelimiter1()}? .+? . ['] {checkQuoteDelimiter2()}?)
    )
;

INT: [0-9]+;

/*----------------------------------------------------------------------------*/
// Identifier
/*----------------------------------------------------------------------------*/

QUOTED_ID: '"' .*? '"' ('"' .*? '"')*;
ID: [\p{Alpha}] [_$#0-9\p{Alpha}]*;

/*----------------------------------------------------------------------------*/
// Islands of interest as single tokens
/*----------------------------------------------------------------------------*/

CALL:
    'call' COMMENT_OR_WS+ SQL_TEXT+? SQL_END
;

DELETE:
    'delete' COMMENT_OR_WS+ SQL_TEXT+? SQL_END
;

EXPLAIN_PLAN:
    'explain' COMMENT_OR_WS+ 'plan' COMMENT_OR_WS+ SQL_TEXT+? SQL_END
;

INSERT:
    'insert' COMMENT_OR_WS+ SQL_TEXT+? SQL_END
;

MERGE:
    'merge' COMMENT_OR_WS+ SQL_TEXT+? SQL_END
;

UPDATE:
    'update' COMMENT_OR_WS+ SQL_TEXT+? 'set' COMMENT_OR_WS+ SQL_TEXT+? SQL_END
;

SELECT:
    (
          ('with' COMMENT_OR_WS+ ('function'|'procedure') SQL_TEXT+? PLSQL_DECLARATION_END)
        | ('with' COMMENT_OR_WS+ SQL_TEXT+? SQL_END)
        | (('(' COMMENT_OR_WS*)* 'select' COMMENT_OR_WS SQL_TEXT+? SQL_END)
    )
;

/*----------------------------------------------------------------------------*/
// Any other token
/*----------------------------------------------------------------------------*/

ANY_OTHER: . -> channel(HIDDEN);

Options

The options are the same as in IslandSqlScopeLexer. We need the superclass IslandSqlLexerBase only in the STRING rule to handle all quote identifiers supported by the Oracle Database.

Fragments

We moved the fragments section to the top. The fragments CONTINUE_LINE, SQLPLUS_TEXT and SQLPLUS_END are not required in this lexer. They are used in IslandSqlLexerBase to identify SQL*Plus commands as hidden tokens.

Since we replaced all SQL*Plus commands with whitespace there is no need to handle SQL*Plus commands in this lexer.

Hidden Tokens

In this section, we define the tokens that we do not need in the parser. Therefore we place them on the hidden channel.

Wait, weren’t the hidden tokens replaced by whitespace? Yes, but only those that were not part of other tokens. However, the statements in scope (represented as a single token in IslandSqlScopeLexer) contain whitespace characters and maybe also comments or even conditional compilation directives (e.g. in plsql_declarations of a select statement). At the current stage of the grammar, the CONDITIONAL_COMPILATION_DIRECTIVE is de facto unused. We will need it (or some adequate replacement) once we are going to implement the select statement or other statements containing PL/SQL code.

Keywords

It’s a good practice to define a rule for each token. This way we can control the names of the constants generated by ANTLR. We prefix the keywords with a K_ to distinguish them from other rules/tokens. These keywords can also be used as identifiers in various contexts. At the current stage of the grammar, this section contains only the keywords used in the lock table statement of the Oracle Database 21c.

Special Characters

The lock table statement uses these special characters.

Data Types

The STRING rule is the same as in IslandSqlScopeLexer. It is a complete definition of a text literal. The INT rule is new. It defines an unsigned integer.

In a future version of the grammar, we will need to support all numeric literals. And of course also date literals and interval literals. We will split the implementation between the lexer and the parser. That will be a bit tricky. For now, let’s keep it simple. – An unsigned integer works in most cases.

Identifier

In the lexer, we define two types of identifiers. Quoted and nonquoted Identifiers. See also Database Object Naming Rules. See also my blog post regarding unnecessary quoted identifiers.

Islands of Interest as Single Token

This is the same list of rules as in IslandSqlScopeLexer. The only rule that is missing is LOCK_TABLE since we are tokenizing this SQL statement completely.

Please note that the UPDATE rule includes a set keyword. This is necessary because the keyword update  is also part of the lock table statement. Without this change, a lock table emp in share update mode nowait; statement would be partly identified as update statement (update mode nowait;).

The final version of the lexer will not contain statements as single tokens.

Any Other Token

As in IslandSqlScopeLexer we put any other character on the hidden channel. This suppresses some errors in the parser. For example, we can insert a euro sign (€) or a pound sign (£) almost anywhere in the code without causing an error.

In future versions of the lexer, we will put the ANY_OTHER token on the DEFAULT_CHANNEL to avoid this kind of error suppression.

Parser Changes

The changes to the previous version of the parser are highlighted.

parser grammar IslandSqlParser;

options {
    tokenVocab=IslandSqlLexer;
}

/*----------------------------------------------------------------------------*/
// Start rule
/*----------------------------------------------------------------------------*/

file: dmlStatement* EOF;

/*----------------------------------------------------------------------------*/
// Data Manipulation Language
/*----------------------------------------------------------------------------*/

dmlStatement:
      callStatement
    | deleteStatement
    | explainPlanStatement
    | insertStatement
    | lockTableStatement
    | mergeStatement
    | selectStatement
    | updateStatement
;

callStatement: CALL;
deleteStatement: DELETE;
explainPlanStatement: EXPLAIN_PLAN;
insertStatement: INSERT;
mergeStatement: MERGE;
updateStatement: UPDATE;
selectStatement: SELECT;

/*----------------------------------------------------------------------------*/
// Lock table
/*----------------------------------------------------------------------------*/

lockTableStatement:
    stmt=lockTableStatementUnterminated sqlEnd
;

lockTableStatementUnterminated:
    K_LOCK K_TABLE objects+=lockTableObject (COMMA objects+=lockTableObject)*
        K_IN lockmode=lockMode K_MODE waitOption=lockTableWaitOption?
;

lockTableObject:
    (schema=sqlName DOT)? table=sqlName
        (
              partitionExtensionClause
            | (AT_SIGN dblink=qualifiedName)
        )?
;

partitionExtensionClause:
      (K_PARTITION OPEN_PAREN name=sqlName CLOSE_PAREN)             # partition
    | (K_PARTITION K_FOR OPEN_PAREN
        (keys+=expression (COMMA keys+=expression)*) CLOSE_PAREN)   # partitionKeys
    | (K_SUBPARTITION OPEN_PAREN name=sqlName CLOSE_PAREN)          # subpartition
    | (K_SUBPARTITION K_FOR OPEN_PAREN
        (keys+=expression (COMMA keys+=expression)*) CLOSE_PAREN)   # subpartitionKeys
;

// TODO: complete according https://github.com/IslandSQL/IslandSQL/issues/11
expression:
      STRING        # stringLiteral
    | INT           # integerLiteral
    | sqlName       # sqlNameExpression
;

lockMode:
      (K_ROW K_SHARE)               # rowShare
    | (K_ROW K_EXCLUSIVE)           # rowExclusive
    | (K_SHARE K_UPDATE)            # shareUpdate
    | (K_SHARE)                     # share
    | (K_SHARE K_ROW K_EXCLUSIVE)   # shareRowExclusive
    | (K_EXCLUSIVE)                 # exclusive
;

lockTableWaitOption:
      K_NOWAIT                  # nowait
    | K_WAIT waitSeconds=INT    # wait
;

/*----------------------------------------------------------------------------*/
// Identifiers
/*----------------------------------------------------------------------------*/

keywordAsId:
      K_EXCLUSIVE
    | K_FOR
    | K_IN
    | K_LOCK
    | K_MODE
    | K_NOWAIT
    | K_PARTITION
    | K_ROW
    | K_SHARE
    | K_SUBPARTITION
    | K_TABLE
    | K_UPDATE
    | K_WAIT
;

unquotedId:
      ID
    | keywordAsId
;

sqlName:
      unquotedId
    | QUOTED_ID
;

qualifiedName:
	sqlName (DOT sqlName)*
;

/*----------------------------------------------------------------------------*/
// SQL statement end, slash accepted without preceeding newline
/*----------------------------------------------------------------------------*/

sqlEnd: EOF | SEMI | SLASH;

Data Manipulation Language

The only visible change in this section is the title. However, there is an important change regarding lockTableStatement.  It’s not a simple rule referring to a lexer token anymore.

Lock Table

lockTableStatement

On lines 40-42 we define the lockTableStatement. It starts with a lockTableStatementUnterminated and ends on sqlEnd. It contains the same number of characters as in the previous parser version. As a result, the extension for Visual Studio Code finds the same lock table statements as before.

lockTableStatementUnterminated

On lines 44-47 we define the lockTableStatementUnterminated according to the Oracle Database SQL Language Reference 21c with the following three fields:

• objects as an array of lockTableObjects with at least one entry
• lockMode that refers to a mandatory instance of lockMode
•  waitOption that refers to an optional instance of lockTableWaitOption

Based on that ANTLR generates a IslandSqlParser class with a nested class LockTableStatementUnterminatedContext.

public class IslandSqlParser extends Parser {
    ...
    public static class LockTableStatementUnterminatedContext extends ParserRuleContext {
        public LockTableObjectContext lockTableObject;
        public List<LockTableObjectContext> objects = new ArrayList<LockTableObjectContext>();
        public LockModeContext lockmode;
        public LockTableWaitOptionContext waitOption;
        ...
    }
    ...
}

The parser populates an instance of LockTableStatementUnterminatedContext according to the input. Interesting is, that there is a redundancy between objects and lockTableObject. The former contains all objects to be locked and the latter just the last one.

Please note that the lock table statement ends on mode keyword or on lockTableWaitOption which can end on wait keyword or on an integer value.

lockTableObject

The lockTableObject on lines 49-55 defines the following fields:

• schema, optional refers to a sqlNameidentifier
• table, mandatory refers to a sqlName identifier
• dblink, optional refers to a qualifiedName identifier

For the optional partitionExtensionClause no field is defined. I think this is wrong and should be fixed in a future version. Nonetheless, it’s possible to find it in the generic children field.

partitionExtensionClause

The partitionExtensionClause on lines 57-64 defines four partition variants. Each variant has a label – the token after the hash sign (#). Based on these labels ANTLR generates the following subclasses of the class PartitionExtensionClauseContext:

• PartitionContext
• SubpartitionKeysContext
• SubpartitionContext
• SubpartitionKeysContext

It’s another good practice to define a label for an alternative. It simplifies finding classes in the parse tree using listeners or visitors and makes the parse tree more expressive. The next screenshots highlight the partition alternative in the ANTLR IntelliJ plugin. The ANTLR interpreter does not generate classes. Instead, it shows the alternative after a colon. Either the ordinal number or the label, if available. However, it’s still a good representation of what you can expect at runtime of the parser generated by ANTLR.

The alternatives for partitionKeys and subpartitionKeys define a field named keys with an array of expression.

expression

When working on a grammar you feel more than once like Hal fixing a light bulb. An expression is probably the most extensive part of the SQL grammar. It’s huge. It contains subqueries and a subquery is basically a select statement and a select statement uses conditions… Once we’ve done that, implementing the rest of the IslandSQL grammar is a piece of cake.

Therefore I decided to postpone the complete implementation and define just the bare minimum on lines 66-71. Making the lock table statement work for partition keys based on integers, strings and variable names. – No datetime expressions yet.

lockMode

The Oracle Database allows 6 different lock modes. You find the valid alternatives on lines 74-79.

lockTableWaitOption

By default, the Oracle Database waits indefinitely for the lock. You can override this behaviour by one of the alternatives defined on lines 83-84.

The grammar defines waitSeconds as an INT. That matches the definition in the SQL Language Reference of the Oracle Database 21c.

However, what is the meaning of integer in this case? Can we use an integer variable in PL/SQL? Can we use a decimal literal that can be converted to an integer such as 10.? Or can we use scientific notations such as 1e2 or even 1e2d? To know that, we have to try it out.

SQL> declare
  2     co_wait_in_seconds constant integer := 10;
  3  begin
  4     lock table emp in exclusive mode wait co_wait_in_seconds;
  5  end;
  6  /
   lock table emp in exclusive mode wait co_wait_in_seconds;
                                         *
ERROR at line 4:
ORA-06550: line 4, column 42:
PL/SQL: ORA-30005: missing or invalid WAIT interval
ORA-06550: line 4, column 4:
PL/SQL: SQL Statement ignored

SQL> lock table emp in exclusive mode wait 10.;

Table(s) Locked.

SQL> lock table emp in exclusive mode wait 1e2;

Table(s) Locked.

SQL> lock table emp in exclusive mode wait 1e2d;
lock table emp in exclusive mode wait 1e2d
                                      *
ERROR at line 1:
ORA-30005: missing or invalid WAIT interval

So, we cannot use a variable/constant. But the scientific notation works and a decimal literal can be converted to an integer as long as we do not use the scientific notation.

I consider it a bug that we cannot use a variable/constant for the time to wait in PL/SQL. Especially since we can use static expressions in PL/SQL in various places, e.g. to define the size of varchar2 variable (since 12.2). It does not make sense to enforce the use of dynamic SQL to handle dynamic wait times in PL/SQL.

I can imagine that a future version of the Oracle Database will lift this restriction in the lock table statement. It would be a small change. Maybe not even documented. Therefore it might be a good idea to change this part of the grammar and support a bit more.

Identifiers

The Oracle Database allows the use of keywords as identifiers in a lot of places. Therefore we should allow the use of keywords such as lock in the lock table statement’s the identifiers schema, table and dblink.  For that, we created a rule named keywordAsId on lines 91-105 that covers all keywords.

We defined ID in the lexer. It covers all identifiers. However, keywords have a higher priority in the lexer. Therefore we defined a new rule unquotedId that combines ID with keywordAsId.

The rule sqlName on lines 112-115 combines unquotedId with the QUOTED_ID which we defined in the lexer.

And finally, the rule qualifiedName on lines 117-119 covers the unbounded concatenation of SqlName with a dot. The concatenation is optional. So a qualifiedName could look 100% the same as a sqlName. We could remove the schema in the rule lockTableObject and use qualifiedName for table like this:

lockTableObject:
    table=qualifiedName
        (
              partitionExtensionClause
            | (AT_SIGN dblink=qualifiedName)
        )?
;

This works and is a valid representation of the grammar. However, it’s less expressive. For dblink we must use qualifiedName.  There is no predefined, binding naming scheme that covers the number of segments for a database link name.

SQL Statement End

The sqlEnd rule on the last line 125 defines the end of a SQL statement in SQL*Plus/SQLcl. We do not handle whitespace here as in the IslandSqlScopeLexer. As a result a lock table statement could be terminated with a slash on the same line. This might need some rework in a future version of the grammar.

Syntax Errors

Let’s look at a lock table  statement that uses an invalid lock mode.

SQL> lock table dept in access exclusive mode;
lock table dept in access exclusive mode
                          *
ERROR at line 1:
ORA-01737: valid modes: [ROW] SHARE, [[SHARE] ROW] EXCLUSIVE, SHARE UPDATE

This is a valid lock mode in PostgresSQL 15. Besides the lock mode, the syntax of the lock table statement is different to the one in the Oracle Database 21c in various places. However, it should not be too complicated to define a grammar that can handle both syntaxes. We put this on the to-do list and focus on supporting the Oracle Database grammar first.

However, how does our grammar deal with invalid SQL statements? – Here’s a screenshot of the extension for Visual Studio Code showing some lock table statements.

You see the word access wavy underlined in red. On mouse over you get the details displayed in the problems panel. Furthermore, you see that lock_table.sql is displayed in red with a number 3 indicating that this file has three problems. And the outline view indicates problems by showing symbols in red.

That’s the cool thing when using an IDE supporting Micosoft’s Language Server Protocol. We just have to provide the syntax errors and the visualization happens automatically by the IDE in a standardized manner.

Right now the parser provides only syntax errors. However, it is relatively easy to implement a linter based on this grammar and provide the results as warnings. For example for lock table statements without a waitOption.

Outlook

We have not succeeded in fully supporting the lock table statement. Some cases cannot yet be successfully parsed. I would like to address that. To do this, we need to look a bit more at literals and expressions before we deal with more DML statements.

Another topic is the support of more SQL dialects.  I’d like to support PostgreSQL. Maybe it is a good time to start as soon as a DML statement is fully covered.

Stay tuned.

The post IslandSQL Episode 3: Lock Table appeared first on Philipp Salvisberg's Blog.

In the last episode, we build the initial version of IslandSQL. An Island grammar for SQL scripts covering select statements. In this blog post, we extend the grammar to handle the remaining DML statements.

The full source code is available on GitHub and the binaries on Maven Central.

Lexer Changes

The lexer grammar contains a new fragment COMMENT_OR_WS on line 98. We use this fragment in all DML lexer rules after the starting keywords. Why? Because we can use comments beside whitespace after a keyword as in with/*comment*/function e_count.... The previous lexer version required a whitespace after the with keyword for select statements with a plsql_declarations clause.

I also merged the former PLSQL_DECLARATION rule into the SELECT rule. Mainly to have a single lexer rule for all DML statements. It’s more consistent and easier to understand IMO.

lexer grammar IslandSqlLexer;

options {
    superClass=IslandSqlLexerBase;
    caseInsensitive = true;
}

/*----------------------------------------------------------------------------*/
// Comments and alike to be ignored
/*----------------------------------------------------------------------------*/

ML_COMMENT: '/*' .*? '*/' -> channel(HIDDEN);
SL_COMMENT: '--' .*? (EOF|SINGLE_NL) -> channel(HIDDEN);

REMARK_COMMAND:
    {isBeginOfCommand()}? 'rem' ('a' ('r' 'k'?)?)?
        (WS SQLPLUS_TEXT*)? SQLPLUS_END -> channel(HIDDEN)
;

PROMPT_COMMAND:
    {isBeginOfCommand()}? 'pro' ('m' ('p' 't'?)?)?
       (WS SQLPLUS_TEXT*)? SQLPLUS_END -> channel(HIDDEN)
;

STRING:
    'n'?
    (
          (['] .*? ['])+
        | ('q' ['] '[' .*? ']' ['])
        | ('q' ['] '(' .*? ')' ['])
        | ('q' ['] '{' .*? '}' ['])
        | ('q' ['] '<' .*? '>' ['])
        | ('q' ['] . {saveQuoteDelimiter1()}? .+? . ['] {checkQuoteDelimiter2()}?)
    ) -> channel(HIDDEN)
;

CONDITIONAL_COMPILATION_DIRECTIVE: '$if' .*? '$end' -> channel(HIDDEN);

/*----------------------------------------------------------------------------*/
// Islands of interest on DEFAULT_CHANNEL
/*----------------------------------------------------------------------------*/

CALL:
    {isBeginOfStatement()}? 'call' COMMENT_OR_WS+ SQL_TEXT+? SQL_END
;

DELETE:
    {isBeginOfStatement()}? 'delete' COMMENT_OR_WS+ SQL_TEXT+? SQL_END
;

EXPLAIN_PLAN:
    {isBeginOfStatement()}? 'explain' COMMENT_OR_WS+ 'plan' COMMENT_OR_WS+ SQL_TEXT+? SQL_END
;

INSERT:
    {isBeginOfStatement()}? 'insert' COMMENT_OR_WS+ SQL_TEXT+? SQL_END
;

LOCK_TABLE:
    {isBeginOfStatement()}? 'lock' COMMENT_OR_WS+ 'table' COMMENT_OR_WS+ SQL_TEXT+? SQL_END
;

MERGE:
    {isBeginOfStatement()}? 'merge' COMMENT_OR_WS+ SQL_TEXT+? SQL_END
;

UPDATE:
    {isBeginOfStatement()}? 'update' COMMENT_OR_WS+ SQL_TEXT+? SQL_END
;

SELECT:
    {isBeginOfStatement()}?
    (
          ('with' COMMENT_OR_WS+ ('function'|'procedure') SQL_TEXT+? PLSQL_DECLARATION_END)
        | ('with' COMMENT_OR_WS+ SQL_TEXT+? SQL_END)
        | (('(' COMMENT_OR_WS*)* 'select' COMMENT_OR_WS SQL_TEXT+? SQL_END)
    )
;

/*----------------------------------------------------------------------------*/
// Whitespace
/*----------------------------------------------------------------------------*/

WS: [ \t\r\n]+ -> channel(HIDDEN);

/*----------------------------------------------------------------------------*/
// Any other token
/*----------------------------------------------------------------------------*/

ANY_OTHER: . -> channel(HIDDEN);

/*----------------------------------------------------------------------------*/
// Fragments to name expressions and reduce code duplication
/*----------------------------------------------------------------------------*/

fragment SINGLE_NL: '\r'? '\n';
fragment CONTINUE_LINE: '-' [ \t]* SINGLE_NL;
fragment COMMENT_OR_WS: ML_COMMENT|SL_COMMENT|WS;
fragment SQLPLUS_TEXT: (~[\r\n]|CONTINUE_LINE);
fragment SQL_TEXT: (ML_COMMENT|SL_COMMENT|STRING|.);
fragment SLASH_END: SINGLE_NL WS* '/' [ \t]* (EOF|SINGLE_NL);
fragment PLSQL_DECLARATION_END: ';'? [ \t]* (EOF|SLASH_END);
fragment SQL_END:
      EOF
    | (';' [ \t]* SINGLE_NL?)
    | SLASH_END
;
fragment SQLPLUS_END: EOF|SINGLE_NL;

Parser Changes

The start rule file on line 11 in the parser grammar is now defined as a unbounded number of dmlStatment. Each DML statement is a single lexer token. It’s still not possible to produce a parse error with this grammar. We only process DML statements. Everything else is hidden and therefore ignored.

parser grammar IslandSqlParser;

options {
    tokenVocab=IslandSqlLexer;
}

/*----------------------------------------------------------------------------*/
// Start rule
/*----------------------------------------------------------------------------*/

file: dmlStatement* EOF;

/*----------------------------------------------------------------------------*/
// Rules for reduced SQL grammar (islands of interest)
/*----------------------------------------------------------------------------*/

dmlStatement:
      callStatement
    | deleteStatement
    | explainPlanStatement
    | insertStatement
    | lockTableStatement
    | mergeStatement
    | selectStatement
    | updateStatement
;

callStatement: CALL;
deleteStatement: DELETE;
explainPlanStatement: EXPLAIN_PLAN;
insertStatement: INSERT;
lockTableStatement: LOCK_TABLE;
mergeStatement: MERGE;
updateStatement: UPDATE;
selectStatement: SELECT;

IslandSQL for VS Code

The extension for Visual Studio Code version 0.2.0 finds text in all DML statements and is not limited to select statements anymore. And a symbol for each DML statement is shown now in the outline view.

Outlook

A grammar that can parse all DML statements sounds like something complete. However, this grammar is far from complete. For code analysis, getting a single token for a DML statement is at best a good starting point.

What we need is a more detailed result. For this, we need to move the logic from the lexer to the parser. In the next episode, we will do this with one of the DML statements.

The post IslandSQL Episode 2: All DML Statements appeared first on Philipp Salvisberg's Blog.

An island grammar focuses only on a small part of a grammar. The island represents the small, interesting part and the sea the rest. In this blog post, I explain the components of an island grammar for SQL scripts named IslandSQL. In the first iteration, we focus on the select statement. Everything else is not of interest for the time being.

Use Case

Let’s assume we want to write an extension for Visual Studio Code that can find text in select statements within SQL files of a workspace. So what is the difference to VSCode’s integrated text search? Well, the text search does not know what a select statement is. It finds occurrences of the search text in all kinds of places. In fact, identifying a select statement is not as easy as one might think.

Let’s look at an example.

/* select * from t1; */
-- select * from t2;
remark select * from t3;
prompt select * from t4;
begin
    sys.dbms_output.put_line('irrelevant: select * from t5;');
end;
/
create or replace procedure p is
begin
   $if false $then irrelevant: select * from t6; $end
   null;
end;
/

This SQL script does not contain relevant select statements. A select statement within a comment is hardly relevant. The same is true for select statements in remark and prompt commands. And I consider select statements within string literals and conditional compilation text also as irrelevant, at least in this example.

So let’s look at another example:

-- simple
select * from dept;
-- subquery_factoring_clause
with
   d as (
      select * from dept
   )
select * from d;
-- plsql_declarations
with
   function e_count (in_deptno in dept.deptno%type) return integer is
      l_count integer;
   begin
      select count(*)
        into l_count
        from emp;
      return l_count;
   end e_count;
select deptno, e_count(deptno)
  from dept
/
-- unterminated
select * from dept

This example script contains four select statements. As you can see, a select statement does not necessarily need to start with the select keyword. Furthermore a select statement can end on semicolon or slash or EOF (end-of-file). In fact, when using plsql_declarations the statement must end on a slash (or EOF).

Here’s a screenshot of the VSCode extension after searching for the regular expression .+ in the demo workspace, highlighting the third search result.

Lexer Grammar

The responsibility of the lexer is to convert a stream of characters to a stream of tokens. We use ANTLR 4 to generate our lexer with Java as the target language.

Here’s the grammar definition for our lexer.

lexer grammar IslandSqlLexer;

options {
    superClass=IslandSqlLexerBase;
    caseInsensitive = true;
}

/*----------------------------------------------------------------------------*/
// Comments and alike to be ignored
/*----------------------------------------------------------------------------*/

ML_COMMENT: '/*' .*? '*/' -> channel(HIDDEN);
SL_COMMENT: '--' .*? (EOF|SINGLE_NL) -> channel(HIDDEN);
REMARK_COMMAND:
    {isBeginOfCommand()}? 'rem' ('a' ('r' 'k'?)?)?
        (WS SQLPLUS_TEXT*)? SQLPLUS_END -> channel(HIDDEN)
;
PROMPT_COMMAND:
    {isBeginOfCommand()}? 'pro' ('m' ('p' 't'?)?)?
       (WS SQLPLUS_TEXT*)? SQLPLUS_END -> channel(HIDDEN)
;
STRING:
    'n'?
    (
          (['] .*? ['])+
        | ('q' ['] '[' .*? ']' ['])
        | ('q' ['] '(' .*? ')' ['])
        | ('q' ['] '{' .*? '}' ['])
        | ('q' ['] '<' .*? '>' ['])
        | ('q' ['] . {saveQuoteDelimiter1()}? .+? . ['] {checkQuoteDelimiter2()}?)
    ) -> channel(HIDDEN)
;
CONDITIONAL_COMPILATION_DIRECTIVE: '$if' .*? '$end' -> channel(HIDDEN);

/*----------------------------------------------------------------------------*/
// Islands of interest on DEFAULT_CHANNEL
/*----------------------------------------------------------------------------*/

PLSQL_DECLARATION:
    {isBeginOfStatement()}? 'with' WS
        ('function'|'procedure') SQL_TEXT*?  PLSQL_DECLARATION_END
;
SELECT:
    {isBeginOfStatement()}? ('with'|('(' WS?)* 'select') SQL_TEXT*? SQL_END
;

/*----------------------------------------------------------------------------*/
// Whitespace
/*----------------------------------------------------------------------------*/

WS: [ \t\r\n]+ -> channel(HIDDEN);

/*----------------------------------------------------------------------------*/
// Any other token
/*----------------------------------------------------------------------------*/

ANY_OTHER: . -> channel(HIDDEN);

/*----------------------------------------------------------------------------*/
// Fragments to name expressions and reduce code duplication
/*----------------------------------------------------------------------------*/

fragment SINGLE_NL: '\r'? '\n';
fragment CONTINUE_LINE: '-' [ \t]* SINGLE_NL;
fragment SQLPLUS_TEXT: (~[\r\n]|CONTINUE_LINE);
fragment SQL_TEXT: (ML_COMMENT|SL_COMMENT|STRING|.);
fragment SLASH_END: SINGLE_NL WS* '/' [ \t]* (EOF|SINGLE_NL);
fragment PLSQL_DECLARATION_END: ';'? [ \t]* (EOF|SLASH_END);
fragment SQL_END:
      EOF
    | (';' [ \t]* SINGLE_NL?)
    | SLASH_END
;
fragment SQLPLUS_END: EOF|SINGLE_NL;

Lexer Options

On lines 3-6 we define the grammar options.

The first option defines the superclass that the generated lexer class should extend from. We use this class to define semantic predicates that can be used in the lexer grammar. Semantic predicates are very powerful. However, they bind a grammar to a target language. Of course, you can implement a superclass in different target languages. But would you want to do that for every supported target language?

The second option defines the grammar as case-insensitive. This simplifies the grammar. We can simply write select instead of S E L E C T where every letter is a fragment (e.g. fragment S: [sS];).

Channels

A token can be either ignored (skipped) or placed on a channel. ANTLR provides by default the following two channels:

• DEFAULT_CHANNEL: for visible tokens that are relevant to the parser grammar
• HIDDEN: for tokens that are not relevant to the parser grammar

We do not skip tokens in this grammar. This has the advantage that we can access hidden tokens when we need to. For example, for accessing hints or for lossless serialisation of chosen parts.

Comments and Alike

On lines 8-33, we define hidden tokens using lexer rules.

The notation for the token definitions should be familiar to those with regular expression experience.
— Terence Parr, The Definitive ANTLR 4 Reference, 2nd edition, page 36

The tokens defined in this section are similar to comments and therefore should be ignored and placed on the hidden channel.

The order of the rules is important in case of conflicting definitions. The first rule wins. ML_COMMENT defines a multiline comment starting with /*  and ending with */. The rules defined afterwards cannot define tokens that are a subset of ML_COMMENT. For example,  a select statement within a ML_COMMENT is not visible for subsequent rules. This is the reason for the rules in this section. We want to hide select statements within comments and alike.

However, it is possible to define a token that contains a ML_COMMENT, e.g. in a select statement.

Islands of Interest

You find the islands of interest on lines 35-45. The rule PLSQL_DECLARATION covers the select statement with a plsql_declarations clause. And the rule SELECT covers the select statement without a plsql_declarations clause. Identifying the end of the select statement is a bit tricky.

This definition works in many cases, but it will falsely match subqueries in other statements such as insert, update, delete, merge, etc. In such cases, everything up to the next semicolon is considered part of the select statement. We will address this flaw in a future version of the grammar.

How do we ignore semicolons in comments and strings? By using the fragment SQL_TEXT. A SQL_TEXT is either a ML_COMMENT, a SL_COMMENT, a STRING or any other character (.). Again, the order is important. The lexer considers comments and strings as single tokens. A semicolon in comments or strings is not visible. As a result, a semicolon in comments or strings will not be interpreted as the end of a select statement.

Did you wonder why I use ANTLR instead of regular expressions? Well, it’s simply not possible to write a regular expression that can match a complete select statement ending on a semicolon while ignoring the semicolons in comments and strings. Simply put, ANTLR is more powerful.

These two rules produce a single token for the whole select statement. This is simple and enough for our current use case. However, in the coming versions of the grammar, we will rewrite this section to parse a select statement completely and address the issues with subqueries in other statements.

Whitespace

The WS rule on line 51 defines whitespace characters. They are not relevant for the parser, hence we put them on the HIDDEN channel.

Other Tokens

The ANY_OTHER rule on line 57 covers any other character. They are not relevant for the parser and we put them also on the HIDDEN channel.

Fragments

And finally, on lines 59-74 we have fragments. A fragment allows one to name an expression and use the fragment name instead of the expression in other fragments or rules. This makes the grammar more readable without introducing additional token types.

Parser Grammar

We use the output of the lexer – the token stream – in the parser. By default, only the tokens on the DEFAULT_CHANNEL are visible in the parser grammar. This makes the grammar quite simple.

parser grammar IslandSqlParser;

options {
    tokenVocab=IslandSqlLexer;
}

/*----------------------------------------------------------------------------*/
// Start rule
/*----------------------------------------------------------------------------*/

file: selectStatement* EOF;

/*----------------------------------------------------------------------------*/
// Rules for reduced SQL grammar (islands of interest)
/*----------------------------------------------------------------------------*/

selectStatement: PLSQL_DECLARATION | SELECT;

Parser Options

On line 4 we include the token vocabulary based on the lexer grammar. The vocabulary defines integer values for each token type, e.g. PLSQL_DECLARATION=7 or SELECT=8. The token stream uses these integer values to identify token types. Integers are shorter than their string counterparts and therefore use fewer resources.

Start Rule

You find the start rule file on line 11. This is the entry point for the parser. The root node of the parse tree.

A file may contain an unbounded number of selectStatement rules. And a file ends on the pseudo tokenEOF (end-of-file). This way we ensure that the parser reads the complete file.

Select Statement

On line 17 the selectStatement is defined either as a PLSQL_DECLARATION or SELECT token.

That’s it. All other tokens are hidden and invisible to the parser.

Furthermore, it’s not possible to produce a parse error with this grammar. Everything that is not a selectStatement is on the hidden channel and irrelevant.

Interpreter

ANTLR is a parser generator. It takes the lexer and parser grammar as input and produces a lexer and parser in a chosen target language as output.

However, there are also ANTLR interpreters. As a plugin in IDEs such as IntelliJ, as a standalone application or as a web application. After pasting the grammers into the UI you can play with it. Here’s a screenshot of the web variant of ANTLR lab.

What about semantic predicates? – The interpreter acts as the call returned true. As a result, for example, xselect * from dual; is falsely recognized as a select statement. Nevertheless, this is usually good enough to explore most parts of an ANTLR grammar.

IslandSQL on GitHub

The source code of the IslandSQL parser is available on GitHub, licensed under the Apache License, Version 2.0.

However, you will not find any test cases in this repo. I have written 92 test cases for the initial version 0.1.0. They are stored in a private repository. I do not plan to release them. It’s a way to make an unfriendly fork harder. Not right now. But once the grammar has evolved to cover a significant subset of SQL of the relevant database management systems, the situation might be different.

IslandSQL on Maven Central

The IslandSQL parser is available on Maven Central. This makes integration into your preferred build system easy.

And using the parser is also easy. Here is an example:

import ch.islandsql.grammar.IslandSqlDocument;
import ch.islandsql.grammar.IslandSqlParser;

class Demo {
    public static void main(String[] args) {
        var doc = IslandSqlDocument.parse("""
                /* select * from t1; */
                -- select * from t2;
                rem select * from t3;
                prompt select * from t4;
                -- simple
                select * from dept;
                -- subquery_factoring_clause
                with d as (select * from dept) select * from d;
                -- other statements
                delete from t5;
                update t6 set c1 = null;
                commit;
                """);
        System.out.println(doc.getFile().getText());
        System.out.println("----------");
        doc.getFile().children.forEach(child -> System.out.print(child.getText()));
        System.out.println("\n----------");
        doc.getAllContentsOfType(IslandSqlParser.SelectStatementContext.class)
                .forEach(stmt -> System.out.print(stmt.getText()));
    }
}

The output of the main method is:

select * from dept;
with d as (select * from dept) select * from d;
<EOF>
----------
select * from dept;
with d as (select * from dept) select * from d;
<EOF>
----------
select * from dept;
with d as (select * from dept) select * from d;

IslandSQL on Visual Studio Code Marketplace

The extension for IslandSQL is available in the Visual Studio Code Marketplace. You can install it directly from any VS Code installation.

Agreed, this extension is currently of limited value. However, it was a good opportunity to learn about VSCode extension development and how to use the tooling around Microsoft’s Language Server Protocol (LSP) to integrate a grammar that is written in Java.

I am sure I can use this knowledge for other projects like utPLSQL once SQL Developer for VScode is available.

Outlook

I plan to extend the IslandSQL grammar step by step and blog about the progress. At some point, it will be necessary to move the logic from the lexer to the parser. Before that, I’ll be working on the lexer side a bit longer.

Adding the missing DML statements to the grammar will be the next item on my to-do list.

Another topic is utPLSQL. The utPLSQL annotations in package specifications could be easily parsed with a dedicated island grammar. We could visualise test suite hierarchies in the IDE and also consider tags. Of course, we would duplicate some of utPLSQL’s code in the database. The advantage of such an approach is that we know where a test package is located in the file system. This helps in navigating to the right place, e.g. after test execution failures and could greatly improve the experience of file-based development (compared to SQL Developer). I am looking forward to the next generation of SQL Developer based on VS Codium, where such an extension would bring the most value.

The post IslandSQL Episode 1: Select Statement appeared first on Philipp Salvisberg's Blog.