IslandSQL is a parser for SQL files targeting OracleDB or PostgreSQL. The parser is available on Maven Central and can process SQL*Plus, SQLcl or psql statements besides SQL statements. However, the focus is on statements with static DML statements and code in PL/SQL and PL/pgSQL. For static code analysis, for example.

In PostgreSQL create function, create procedure and do accept code as a string. This simplifies parsing and the implementation of additional languages. As a result, you can write functions and procedures in SQL, PL/pgSQL, PL/Tcl, PL/Perl and PL/Python in any standard PostgreSQL distribution.

Starting with IslandSQL version 0.10 it’s possible to parse SQL and PL/pgSQL in strings and extend the parse tree accordingly. In this blog post, I will explain how this works.

This VS Code extension uses IslandSQL in a language server to report syntax errors and to produce parse trees as shown in this blog post.

PL/pgSQL as String

Let’s look at a do statement executing a PL/pgSQL block provided as a string.

do '
begin
   raise notice $$Hello World!$$;
end
';

NOTICE:  Hello World!
DO

The parse tree in IslandSQL version 0.9 looks as follows:

Look at the parse tree. It is quite simple. Interesting is the PL/pgSQL block. The content in single quotes is represented as a single token. The big rectangle at the bottom. It’s a single token regardless of the code size.

PL/pgSQL as Subtree

Parsing PL/pgSQL as a string is easy. The lexer produces the token for the string containing the PL/pgSQL code. The parser does not need to understand the PL/pgSQL at all.

But this does not help analyse the PL/pgSQL code. We need a parser that

• understands PL/pgSQL
• parses PL/pgSQL code provided as a string
• extends the main parse tree by sub-parse trees with PL/pgSQL code

The IslandSQL parser in version 0.10 does exactly that. It produces this parse tree by default:

The PL/pgSQL code is represented twice in this parse tree. Once as a single token. And once as a node named postgresqlPlpgsqlCode.

PL/pgSQL as Subtree Only

Maybe you do not like it when PL/pgSQL code is represented twice in the parse tree. In this case, you can override the default options when creating an IslandSQL document.

The IslandSQL library requires Java 8 or newer. In the next example, we use Java 22 to reduce boilerplate code. Implicitly declared classes are still in preview. Hence we have to pass the parameters --enable-preview --source 22 when running the HelloWorld.java program.

Explaining HelloWorld.java

In line 13 we create the IslandSQL document with a series of parameters. One of them is removeCode(true) in line 19 which oversteers the default behaviour and removes the code as a string from the parse tree.

In line 21 we print a profile. Thanks to Cary Millsap, I can’t deal with performance issues without thinking about a bicycle, a Ferrari and kissing… The profile helps to understand where the parser has spent its time.

Finally, in lines 22 and 23 we print the parse tree. Once as a simple textual hierarchy and once as a DOT graph. You can visualise the result in Edotor.net, for example.

import ch.islandsql.grammar.IslandSqlDialect;
import ch.islandsql.grammar.IslandSqlDocument;
import ch.islandsql.grammar.util.ParseTreeUtil;

void main() {
    var source = """
            do '
            begin
               raise notice $$Hello World!$$;
            end
            ';
            """;
    var doc = new IslandSqlDocument.Builder()
            .sql(source)
            .hideOutOfScopeTokens(false)
            .dialect(IslandSqlDialect.POSTGRESQL)
            .profile(true)
            .subtrees(true)
            .removeCode(true)
            .build();
    System.out.println(doc.getParserMetrics().printProfile());
    System.out.println(ParseTreeUtil.printParseTree(doc.getFile()));
    System.out.println(ParseTreeUtil.dotParseTree(doc.getFile()));
}

java --enable-preview --source 22 -cp .:islandsql-0.10.0.jar:antlr4-runtime-4.13.1.jar HelloWorld.java

Profile
=======

Total memory used by parser    : 5’164 KB
Total time spent in parser     : 27.918 ms
Total time recorded by profiler: 14.167 ms (100%)

Rule Name (Decision)                          Time (ms) Percent Invocations Lookahead Max Lookahead Ambiguities Errors
---------------------------------------- -------------- ------- ----------- --------- ------------- ----------- ------
string (1887)                                    10.302   72.72           2         0             0           0      0
plsqlStatement (1129)                             1.337    9.44           1         0             0           0      0
postgresqlDo (576)                                0.750    5.30           1         0             0           0      0
postgresqlPlpgsqlCode (6)                         0.735    5.19           2         0             0           0      0
statement (11)                                    0.566    3.99           1         0             0           0      0
sqlEnd (1888)                                     0.314    2.21           1         0             0           0      0
postgresqlRaiseStatement (1260)                   0.134    0.95           1         0             0           0      0
postgresqlRaiseStatement (1269)                   0.030    0.21           1         0             0           0      0

file
  statement
    doStatement
      postgresqlDo
        K_DO:do
        postgresqlPlpgsqlCode
          K_BEGIN:begin
          plsqlStatement
            postgresqlRaiseStatement
              K_RAISE:raise
              raiseLevel
                K_NOTICE:notice
              string:dollarString
                DOLLAR_STRING:$$Hello World!$$
              SEMI:;
          K_END:end
      sqlEnd
        SEMI:;
  <EOF>

digraph islandSQL {
  bgcolor="transparent"
  "1155757579" [shape=ellipse label="file" style=filled fillcolor="#bfe6ff" fontname="Helvetica"]
  "1155757579" -> "1785111044"
  "1155757579" -> "1482748887"
  "1785111044" [shape=ellipse label="statement" style=filled fillcolor="#bfe6ff" fontname="Helvetica"]
  "1785111044" -> "494894055"
  "494894055" [shape=ellipse label="doStatement" style=filled fillcolor="#bfe6ff" fontname="Helvetica"]
  "494894055" -> "1123226989"
  "494894055" -> "500885941"
  "1123226989" [shape=ellipse label="postgresqlDo" style=filled fillcolor="#bfe6ff" fontname="Helvetica"]
  "1123226989" -> "1115381650"
  "1123226989" -> "616412281"
  "1115381650" [shape=box label="do" style=filled fillcolor="#fadabd" fontname="Helvetica"]
  "616412281" [shape=ellipse label="postgresqlPlpgsqlCode" style=filled fillcolor="#bfe6ff" fontname="Helvetica"]
  "616412281" -> "2118096382"
  "616412281" -> "878861517"
  "616412281" -> "746394140"
  "2118096382" [shape=box label="begin" style=filled fillcolor="#fadabd" fontname="Helvetica"]
  "878861517" [shape=ellipse label="plsqlStatement" style=filled fillcolor="#bfe6ff" fontname="Helvetica"]
  "878861517" -> "1705665942"
  "1705665942" [shape=ellipse label="postgresqlRaiseStatement" style=filled fillcolor="#bfe6ff" fontname="Helvetica"]
  "1705665942" -> "1731763384"
  "1705665942" -> "1100619942"
  "1705665942" -> "87242619"
  "1705665942" -> "864248990"
  "1731763384" [shape=box label="raise" style=filled fillcolor="#fadabd" fontname="Helvetica"]
  "1100619942" [shape=ellipse label="raiseLevel" style=filled fillcolor="#bfe6ff" fontname="Helvetica"]
  "1100619942" -> "285074186"
  "285074186" [shape=box label="notice" style=filled fillcolor="#fadabd" fontname="Helvetica"]
  "87242619" [shape=ellipse label="dollarString" style=filled fillcolor="#bfe6ff" fontname="Helvetica"]
  "87242619" -> "15892131"
  "15892131" [shape=box label="$$Hello World!$$" style=filled fillcolor="#fadabd" fontname="Helvetica"]
  "864248990" [shape=box label=";" style=filled fillcolor="#fadabd" fontname="Helvetica"]
  "746394140" [shape=box label="end" style=filled fillcolor="#fadabd" fontname="Helvetica"]
  "500885941" [shape=ellipse label="sqlEnd" style=filled fillcolor="#bfe6ff" fontname="Helvetica"]
  "500885941" -> "484841769"
  "484841769" [shape=box label=";" style=filled fillcolor="#fadabd" fontname="Helvetica"]
  "1482748887" [shape=box label="<EOF>" style=filled fillcolor="#fadabd" fontname="Helvetica"]
}

The parse tree looks now like this:

SQL Dialect

In line 16 of the previous HelloWorld.java program we set the dialect to POSTGRESQL. Is that required? – No, it’s not. But when do we need to set the SQL dialect in IslandSQL? – When the lexical incompatibility between OracleDB and PostgreSQL leads to syntax errors in the code to be parsed.

What? – Let me explain.

Identifiers

OracleDB and PostgreSQL use different characters to build an identifier. The following table shows the differences. The allowed characters are listed in square brackets. Read \p{Alpha} as any alphabetic letter in the character set of the database.

PostgreSQL allows identifiers that start with an underscore. That’s not a problem. However, OracleDB allows the hash sign (#) to be used in an identifier. That leads to unexpected results when the PostgreSQL code uses the bitwise XOR operator without spaces around it. Here’s an example

select a#b from t;

PostgreSQL expects that the columns a and b exists in table t.

OracleDB expects that the column a#b exists in table t.

In this case, IslandSQL can parse the code without errors. However, the parse tree might not look as expected. That’s a documented limitation and cannot be influenced by setting the dialect. At least not in version 0.10 of IslandSQL. Nevertheless, it shows the impact of a lexical incompatibility.

Inquiry Directives

PL/SQL supports predefined and custom Inquiry Directives. These directives are lexically incompatible with PostgreSQL dollar-quoted string constants.

Here’s an example:

alter session set plsql_ccflags = 'custom1:41, custom2:42';
begin
   dbms_output.put_line($$custom1);
   dbms_output.put_line($$custom2 || '(2)');
end;

By default, this causes a parse error, because $$custom1);\n dbms_output.put_line($$ is identified as a dollar-quoted string constant by the lexer. As a result, the code is interpreted like this:

alter session set plsql_ccflags = 'custom1:41, custom2:42';
begin
   dbms_output.put_line('custom1);
   dbms_output.put_line('custom2 || '(2)');
end;

This makes it clearer, why we get a syntax error in line 4 at custom2.

In this case, we have to set the SQL dialect to ORACLEDB to parse custom_inquiry_directives.sql without errors.

Predefined Inquiry Directives

To simplify the use, we want to avoid specifying the SQL dialect. One way to achieve that is to handle predefined inquiry directives in the GENERIC SQL dialect.

Here’s an example, that does not report syntax errors:

begin
   dbms_output.put_line($$plsql_line);
   dbms_output.put_line($$plsql_line || '(2)');
end;

We know the predefined inquiry directives and can deal with them in the lexer.

However, this special treatment can cause problems in corner cases like this one:

do '
begin
   raise notice $$plsql_line is a predefined inquiry directive$$;
end
';

In such cases, we should use the POSTGRESQL dialect to avoid parse errors.

Detect SQL Dialect

Is there a way to detect the SQL dialect of an SQL input automatically? Sure. Several. I’m sure there is a way to use an LLM to get a reasonable result. I’m more of a rule-based guy. So we could parse the code with one dialect and on parse errors try other dialects. If all dialects produce errors, we could choose the one with the fewest errors.

This sounds costly, right? Therefore I decided to start with a simple SQL dialect detection mechanism. The current implementation looks like this:

private static IslandSqlDialect guessDialect(String sql) {
    return sql.contains("\n/\n") ? IslandSqlDialect.ORACLEDB : IslandSqlDialect.GENERIC;
}

In other words, if the code contains a slash followed by a newline character in the first column of a line, then we go with the ORACLEDB dialect. In all other cases, we go with the GENERIC SQL dialect.

My first version was even simpler. However, I had to add a final \n to the search string to ensure that SQL code containing multiline comments is not recognized as ORACLEDB dialect. Files with Windows newline characters are always recognized as GENERIC. That’s the price when trying to keep things simple and fast.

Now, the ORACLEDB SQL dialect is correctly detected in the next example. As a result, no syntax errors are reported.

alter session set plsql_ccflags = 'custom1:41, custom2:42';
begin
   dbms_output.put_line($$custom1);
   dbms_output.put_line($$custom2 || '(2)');
end;
/

The SQL dialect detection mechanism kicks in when no SQL dialect is specified (null).

Conclusion

Developing a single grammar for OracleDB and PostgreSQL was an interesting work. I learned a lot about the underlying DBMS. I often looked at the grammar documentation and did not understand it fully. So I had to run the provided examples or create some myself. The typical test cases are based on working examples, extended by tests based on the cartesian product of a subset of clauses that verified if I had defined the order, the optionality, and the cardinality according to the documentation.

A challenge is the undocumented stuff. There are various reasons why something is not documented. In the end, it does not matter why something is missing. The parser fails when processing code that works but should not according to the docs. This cannot be covered by tests based on the official documentation. I found some bugs while processing real-life code. And I expect to find more.

This is the season finale of IslandSQL. There won’t be a second season. However, there might be some spin-offs since I plan to build products based on IslandSQL. Therefore I plan to keep the parser compatible with the latest versions of OracleDB and PostgreSQL.

Feedback is welcome. Please leave your comments on this blog post or open an issue in the IslandSQL GitHub repository for questions, bugs or feature requests.

Thank you.

The post IslandSQL Final Episode 10: Parsing PL/pgSQL appeared first on Philipp Salvisberg's Blog.

In the last episode, we looked at some new features in Oracle Database 23.4. The IslandSQL grammar now covers all statements that can contain static DML statements and code in PL/SQL and PL/pgSQL.

While implementing the ANTLR grammar for the create JSON relational duality view statement I stumbled over GraphQL in this syntax diagram:

You can use GraphQL as an alternative to a subquery to describe the source of your JSON relational duality view. This feature was already part of 23.3. Usually, I don’t like it when there are several ways to do the same thing. However, in this case, GraphQL helped me to understand the variant of the select statement in the duality view better. GraphQL might even be better suited to describe the content of a duality view.

The funny thing is that JSON is about schema flexibility and GraphQL needs a schema to work. That sounds contradictory. However, if we reduce the scope of schema flexibility to a JSON object within existing entities, this can work quite well.

In this blog post, I explore some features related to the schema flexibility of JSON relational duality views.

1. Setup
2. Read-only View à la 19c
3. Read-only Duality View
4. Updateable Duality View Using SELECT
5. Updateable Duality View Using GraphQL
6. GraphQL vs. SELECT
7. Insert Into Duality View
8. Update Duality View
9. Delete From Duality View

1. Setup

The examples in this blog post require an Oracle Database 23.4 (yes, 23.3 is not enough). In a schema of your choice you can run the following setup script:

set linesize 200
set pagesize 1000  
set long 32767
column ext format a72
column data format a130
alter session set nls_date_format = 'YYYY-MM-DD';

drop table if exists emp;
drop table if exists dept;

create table dept (
   deptno number(2, 0)      not null constraint dept_pk primary key,
   dname  varchar2(14 char) not null,
   loc    varchar2(13 char) not null,
   ext    json(object)
);

create table emp (
   empno    number(4, 0)      not null  constraint emp_pk primary key,
   ename    varchar2(10 char) not null,
   job      varchar2(9 char)  not null,
   mgr      number(4, 0)                constraint emp_mgr_fk references emp,
   hiredate date              not null,
   sal      number(7, 2)      not null,
   comm     number(7, 2),
   deptno   number(2, 0)      not null  constraint emp_deptno_fk references dept,
   ext      json(object)
);

insert into dept (deptno, dname, loc)
values (10, 'ACCOUNTING', 'NEW YORK'),
       (20, 'RESEARCH',   'DALLAS'),
       (30, 'SALES',      'CHICAGO'),
       (40, 'OPERATIONS', 'BOSTON');
commit;
       
insert into emp (empno, ename, job, mgr, hiredate, sal, comm, deptno)
values (7566, 'JONES',  'MANAGER',   7839, date '1981-04-02', 2975, null, 20),
       (7698, 'BLAKE',  'MANAGER',   7839, date '1981-05-01', 2850, null, 30),
       (7782, 'CLARK',  'MANAGER',   7839, date '1981-06-09', 2450, null, 10),
       (7788, 'SCOTT',  'ANALYST',   7566, date '1987-04-19', 3000, null, 20),
       (7902, 'FORD',   'ANALYST',   7566, date '1981-12-03', 3000, null, 20),
       (7499, 'ALLEN',  'SALESMAN',  7698, date '1981-02-20', 1600,  300, 30),
       (7521, 'WARD',   'SALESMAN',  7698, date '1981-02-22', 1250,  500, 30),
       (7654, 'MARTIN', 'SALESMAN',  7698, date '1981-09-28', 1250, 1400, 30),
       (7844, 'TURNER', 'SALESMAN',  7698, date '1981-09-08', 1500,    0, 30),
       (7900, 'JAMES',  'CLERK',     7698, date '1981-12-03',  950, null, 30),
       (7934, 'MILLER', 'CLERK',     7782, date '1982-01-23', 1300, null, 10),
       (7369, 'SMITH',  'CLERK',     7902, date '1980-12-17',  800, null, 20),
       (7839, 'KING',   'PRESIDENT', null, date '1981-11-17', 5000, null, 10),
       (7876, 'ADAMS',  'CLERK',     7788, date '1987-05-23', 1100, null, 20);
commit;

Session altered.


Table EMP dropped.


Table DEPT dropped.


Table DEPT created.


Table EMP created.


4 rows inserted.


Commit complete.


14 rows inserted.


Commit complete.

There are a few changes to the well-known dept and emp tables I’d like to highlight:

• Firstly, primary keys and foreign keys. They are required for the duality views, however, they do not need to be enabled.
• Secondly, both tables got an additional ext column. The data type is json(object). Before 23.4 there was just a generic json data type. With 23.4 it’s possible to add modifiers. The modifier object is required for flex columns in duality views.

2. Read-Only View à la 19c

Let’s step back and create a view that returns a single JSON column, as in Oracle Database 19c (to make it work in 19c you have to use for example ext clob check (ext is json) instead of ext json(object) in the tables dept and emp).

create or replace view dept_v as
select json_object(
          deptno,
          dname,
          loc,
          ext,
          'sal': (select sum(sal) from emp where emp.deptno = dept.deptno)
          absent on null
       ) as data
  from dept;

select * from dept_v;

View DEPT_V created.


DATA
--------------------------------------------------------------
{"deptno":10,"dname":"ACCOUNTING","loc":"NEW YORK","sal":8750}
{"deptno":20,"dname":"RESEARCH","loc":"DALLAS","sal":10875}
{"deptno":30,"dname":"SALES","loc":"CHICAGO","sal":9400}
{"deptno":40,"dname":"OPERATIONS","loc":"BOSTON"}

3. Read-Only Duality View

And now let’s try to use the previous subquery in a duality view.

create or replace json duality view dept_dv as
select json_object(
          deptno,
          dname,
          loc,
          ext,
          'sal': (select sum(sal) from emp where emp.deptno = dept.deptno)
          absent on null
       ) as data
  from dept;

ORA-40616: Cannot create JSON Relational Duality View 'DEPT_DV': using ABSENT ON NULL in JSON_OBJECT() is not permitted.

This does not work. The absent on null clause on line 8 is not supported in a duality view. Let’s remove this line and try again.

create or replace json duality view dept_dv as
select json_object(
          deptno,
          dname,
          loc,
          ext,
          'sal': (select sum(sal) from emp where emp.deptno = dept.deptno)
       ) as data
  from dept;

ORA-40895: invalid SQL expression in JSON relational duality view (operators except JSON_OBJECT or JSON_ARRAYAGG not allowed)

This still does not work. It’s not allowed to include an aggregate as in line 7. Let’s also remove this line and try again.

create or replace json duality view dept_dv as
select json_object(
          deptno,
          dname,
          loc,
          ext
       ) as data
  from dept;

ORA-40941: cannot specify a column name or subquery alias for JSON relational duality view

Argh. We cannot use the column alias data on line 7. Let’s remove the alias and try again.

create or replace json duality view dept_dv as
select json_object(
          deptno,
          dname,
          loc,
          ext
       )
  from dept;

ORA-42647: Missing '_id' field at the root level for JSON-relational duality view 'DEPT_DV'.

Okay, an _id field is required to identify a document. Composite keys can be passed as a JSON array. In this case, we do not need that. We can use deptno. Let’s amend the query and try again.

create or replace json duality view dept_dv as
select json_object(
          '_id': deptno,
          dname,
          loc,
          ext
       )
  from dept;

select * from dept_dv;

Json DUALITY created.


DATA
----------------------------------------------------------------------------------------------------------------------------------
{"_id":10,"dname":"ACCOUNTING","loc":"NEW YORK","_metadata":{"etag":"38CBB37294BEE09C6D9867B5B1871FE2","asof":"000025652FBC556F"}}
{"_id":20,"dname":"RESEARCH","loc":"DALLAS","_metadata":{"etag":"1D1973E9B068183129F7DA59F6A9C283","asof":"000025652FBC556F"}}
{"_id":30,"dname":"SALES","loc":"CHICAGO","_metadata":{"etag":"11CC2AE352D52FFDAE0B7A3DFC99F836","asof":"000025652FBC556F"}}
{"_id":40,"dname":"OPERATIONS","loc":"BOSTON","_metadata":{"etag":"B33BBA9A74C046813C59BA0763AD81C9","asof":"000025652FBC556F"}}

Finally, we have a working read-only duality view.

Please note that each document contains a _metadata object. The etag field is a hash value based on all document fields by default. It can be used for optimistic locking. The asof field represents the SCN (system change number). It’s useful for read consistency, this means querying related data in subsequent queries using the flashback_query_clause.

4. Updateable Duality View Using SELECT

Let’s create an updateable duality view that represents all data in our model and uses all relationships.

The highlighted lines contain clauses that work only in a duality view. In other words, the “select” part does not work as a standalone statement as in common relational views.

create or replace json duality view dept_dv as
select json {
          '_id': deptno,
          dname,
          loc,
          ext as flex,
          'emps':
             (
                select json_arrayagg(
                          JSON {
                             emp.empno,
                             emp.ename,
                             emp.job,
                             unnest
                                (
                                   select json {
                                             'mgr'    : mgr.empno with nocheck,
                                             'mgrname': mgr.ename with nocheck
                                          }
                                     from emp mgr
                                    where mgr.empno = emp.mgr
                                ),
                             emp.hiredate,
                             emp.sal,
                             emp.comm,
                             ext as flex
                          }
                       )
                  from emp with insert update delete
                 where emp.deptno = dept.deptno
             )
       }
  from dept with insert update delete;

select json_serialize(data returning clob pretty) as data
  from dept_dv dv
 where dv.data."_id".numberOnly() in (20, 40);

Json DUALITY created.


DATA
------------------------------------------------
{
  "_id" : 20,
  "_metadata" :
  {
    "etag" : "88A4A9648C2CA1752E477545DCA85FD3",
    "asof" : "00002565300350E5"
  },
  "dname" : "RESEARCH",
  "loc" : "DALLAS",
  "emps" :
  [
    {
      "empno" : 7369,
      "ename" : "SMITH",
      "job" : "CLERK",
      "mgr" : 7902,
      "mgrname" : "FORD",
      "hiredate" : "1980-12-17T00:00:00",
      "sal" : 800,
      "comm" : null
    },
    {
      "empno" : 7566,
      "ename" : "JONES",
      "job" : "MANAGER",
      "mgr" : 7839,
      "mgrname" : "KING",
      "hiredate" : "1981-04-02T00:00:00",
      "sal" : 2975,
      "comm" : null
    },
    {
      "empno" : 7788,
      "ename" : "SCOTT",
      "job" : "ANALYST",
      "mgr" : 7566,
      "mgrname" : "JONES",
      "hiredate" : "1987-04-19T00:00:00",
      "sal" : 3000,
      "comm" : null
    },
    {
      "empno" : 7876,
      "ename" : "ADAMS",
      "job" : "CLERK",
      "mgr" : 7788,
      "mgrname" : "SCOTT",
      "hiredate" : "1987-05-23T00:00:00",
      "sal" : 1100,
      "comm" : null
    },
    {
      "empno" : 7902,
      "ename" : "FORD",
      "job" : "ANALYST",
      "mgr" : 7566,
      "mgrname" : "JONES",
      "hiredate" : "1981-12-03T00:00:00",
      "sal" : 3000,
      "comm" : null
    }
  ]
}

{
  "_id" : 40,
  "_metadata" :
  {
    "etag" : "28E9C49240CD26A29FDED7B253A38ED7",
    "asof" : "00002565300350E5"
  },
  "dname" : "OPERATIONS",
  "loc" : "BOSTON",
  "emps" :
  [
  ]
}

Since the ext column in the dept and emp table is empty for all rows, we do not see any additional fields in the two JSON documents.

Due to the unest clause in line 14, the fields mgr and mgrname appear on the same level as all other fields of the table emp.

5. Updateable Duality View Using GraphQL

The next duality view is equivalent to the one in the previous chapter.

The highlighted lines with annotations match the highlighted clauses in the previous statement.

GraphQL requires a model for a query. The Oracle implementation uses tables, primary, and foreign keys to build the underlying model.

Some explanations

• In line 2, we use the dept table as root. For select, insert, update and delete. This means we get a JSON document per department.
• In line 8, we use the emp table for the field emps. We expect an array of objects. However, we do not have to tell that explicitly. The Oracle Database will figure that out. There is just one relationship between dept and emp. Therefore it is clear how to join the tables and access the data. For select, insert, update and delete.
• In line 13, we use the emp table for the fields mgr and mgrname. The access is possible via the foreign keys emp_mgr_fk and emp_deptno_fk. We know that it is emp_mgr_fk but the Oracle Database does not. We have to tell it. We do that with the @link(from: [mgr]) annotation, to use the mgr field for the recursive join. An update of mgrname is not allowed (it is read-only by default). An update of mgr is allowed (it will update the foreign key column in emp).

create or replace json duality view dept_dv as
dept @insert @update @delete
{
   _id: deptno
   dname
   loc
   ext @flex
   emps: emp @insert @update @delete
      {
         empno
         ename
         job
         emp @unnest @link(from: [mgr])
            {
               mgr    : empno @nocheck
               mgrname: ename @nocheck
            }
         hiredate
         sal
         comm
         ext @flex
      }
};

select json_serialize(data returning clob pretty) as data
  from dept_dv dv
 where dv.data."_id".numberOnly() in (20, 40);

Json DUALITY created.


DATA
------------------------------------------------
{
  "_id" : 20,
  "_metadata" :
  {
    "etag" : "88A4A9648C2CA1752E477545DCA85FD3",
    "asof" : "00002565301D6D5C"
  },
  "dname" : "RESEARCH",
  "loc" : "DALLAS",
  "emps" :
  [
    {
      "empno" : 7369,
      "ename" : "SMITH",
      "job" : "CLERK",
      "mgr" : 7902,
      "mgrname" : "FORD",
      "hiredate" : "1980-12-17T00:00:00",
      "sal" : 800,
      "comm" : null
    },
    {
      "empno" : 7566,
      "ename" : "JONES",
      "job" : "MANAGER",
      "mgr" : 7839,
      "mgrname" : "KING",
      "hiredate" : "1981-04-02T00:00:00",
      "sal" : 2975,
      "comm" : null
    },
    {
      "empno" : 7788,
      "ename" : "SCOTT",
      "job" : "ANALYST",
      "mgr" : 7566,
      "mgrname" : "JONES",
      "hiredate" : "1987-04-19T00:00:00",
      "sal" : 3000,
      "comm" : null
    },
    {
      "empno" : 7876,
      "ename" : "ADAMS",
      "job" : "CLERK",
      "mgr" : 7788,
      "mgrname" : "SCOTT",
      "hiredate" : "1987-05-23T00:00:00",
      "sal" : 1100,
      "comm" : null
    },
    {
      "empno" : 7902,
      "ename" : "FORD",
      "job" : "ANALYST",
      "mgr" : 7566,
      "mgrname" : "JONES",
      "hiredate" : "1981-12-03T00:00:00",
      "sal" : 3000,
      "comm" : null
    }
  ]
}

{
  "_id" : 40,
  "_metadata" :
  {
    "etag" : "28E9C49240CD26A29FDED7B253A38ED7",
    "asof" : "00002565301D6D5C"
  },
  "dname" : "OPERATIONS",
  "loc" : "BOSTON",
  "emps" :
  [
  ]
}

The result is the same as for the variant based on select. The only difference is the asof fields, which is expected.

6. GraphQL vs. SELECT

What I like about the GraphQL variant is that the syntax is simple. It looks similar to the result document and is easy to read. No compromises due to feature parity. I run less risk of trying SQL expressions that are not applicable in a duality view. The downside is that the definition might become ambiguous when extending the model with additional foreign key relationships. You might need to add @link annotations to your existing duality views to successfully recreate them. The variant using select cannot become ambiguous. There is no default join logic in SQL yet.

However, writing complex duality views might be easier with the variant using select. I can temporarily comment out all duality-view-specific clauses to make the select part work as a standalone statement until I’m happy with the result.

From a performance point of view, it should theoretically not matter which syntax variant you use. Any duality view can be built on GraphQL or select. The optimizer has all the information it needs to produce an optimal execution plan for both variants. I see no reason why the internal representation should differ.

Maybe a future version of the Oracle Database will offer options to generate the preferred syntax variant independently of the originally deployed variant. By extending dbms_metadata.get_ddl, for example.

7. Insert Into Duality View

Here’s an example of inserting a JSON document with one department and two employees into the previously created duality view.

Some explanations

• In line 6, we populate a department field named secret with the boolean value true. This field does not exist in the model. Therefore it will be stored in the ext column of the dept table.
• In line 13, we set the mgr field to 1. That’s the foreign key column in the emp table.
• In line 16 we populate an employee field named tools with an array. The field does not exist in the model. Therefore it will be stored in the ext column of the emp table.

insert into dept_dv values ('
{
  "_id" : 50,
  "dname" : "MI6",
  "loc" : "LONDON",
  "secret" : true,
  "emps" :
  [
    {
      "empno" : 7,
      "ename" : "BOND",
      "job" : "AGENT",
      "mgr" : 1,
      "hiredate" : "1950-01-01T00:00:00",
      "sal" : 500,
      "tools" : ["Knife", "Garrote Watch", "Walther PPK"]
    },
    {
      "empno" : 1,
      "ename" : "M",
      "job" : "MANAGER",
      "hiredate" : "1940-01-01T00:00:00",
      "sal" : 1000,
      "comm" : 8000
    }
  ]
}
');
commit;

select * from dept where deptno = 50;
select * from emp where deptno = 50;
select json_serialize(data returning clob pretty) as data 
  from dept_dv dv
 where dv.data.secret.booleanOnly();

1 row inserted.


Commit complete.


    DEPTNO DNAME          LOC           EXT
---------- -------------- ------------- --------------------
        50 MI6            LONDON        {"secret":true}


     EMPNO ENAME      JOB              MGR HIREDATE          SAL       COMM     DEPTNO EXT
---------- ---------- --------- ---------- ---------- ---------- ---------- ---------- ---------------------------------------------------
         1 M          MANAGER              1940-01-01       1000       8000         50
         7 BOND       AGENT              1 1950-01-01        500                    50 {"tools":["Knife","Garrote Watch","Walther PPK"]}


DATA
------------------------------------------------
{
  "_id" : 50,
  "_metadata" :
  {
    "etag" : "486256AA33D638F4D339FFE534EC910F",
    "asof" : "0000256530950ADB"
  },
  "dname" : "MI6",
  "loc" : "LONDON",
  "emps" :
  [
    {
      "empno" : 1,
      "ename" : "M",
      "job" : "MANAGER",
      "hiredate" : "1940-01-01T00:00:00",
      "sal" : 1000,
      "comm" : 8000
    },
    {
      "empno" : 7,
      "ename" : "BOND",
      "job" : "AGENT",
      "mgr" : 1,
      "mgrname" : "M",
      "hiredate" : "1950-01-01T00:00:00",
      "sal" : 500,
      "comm" : null,
      "tools" :
      [
        "Knife",
        "Garrote Watch",
        "Walther PPK"
      ]
    }
  ],
  "secret" : true
}

One insert statement leads to three new rows in two tables. Before 23ai, a view and an instead-of trigger would have been required for this.

The fields secret and tools are automatically stored in the flex columns ext. This shows how easy it is to extend the data model on the fly with an insert statement. Without DDL statements. Without PL/SQL code.

8. Update Duality View

Let’s update the previously created document.

Some explanations

• In line 4, we add a new field named street for the department 50.
• In line 6, we change the salary for all employees in the department 50 by the factor of 42.
• In line 9, we increase the salary of BOND by 1. Please note that this is the second change of the salary for this employee in this update statement.
• In line 10, we append the Aston Martin DB5 to the list of BOND‘s tools.

update dept_dv v
   set v.data = json_transform(
                   v.data, 
                   set '$.street' = '85 Albert Embankment',
                   nested '$.emps[*]' (
                      set '@.sal' = path '@.sal * 42'
                   ),
                   nested '$.emps[*]?(@.ename == "BOND")' (
                      set '@.sal' = path '@.sal + 1',
                      append '@.tools' = 'Aston Martin DB5'
                   )
                )
 where v.data."_id".numberOnly() = 50;
commit;

select * from dept where deptno = 50;
select * from emp where deptno = 50;
select json_serialize(data returning clob pretty) as data 
  from dept_dv v
 where v.data."_id".numberOnly() = 50;

1 row updated.


Commit complete.


    DEPTNO DNAME          LOC           EXT
---------- -------------- ------------- -----------------------------------------------
        50 MI6            LONDON        {"secret":true,"street":"85 Albert Embankment"}


     EMPNO ENAME      JOB              MGR HIREDATE          SAL       COMM     DEPTNO EXT
---------- ---------- --------- ---------- ---------- ---------- ---------- ---------- ----------------------------------------------------------------------
         1 M          MANAGER              1940-01-01      42000       8000         50
         7 BOND       AGENT              1 1950-01-01      21001                    50 {"tools":["Knife","Garrote Watch","Walther PPK","Aston Martin DB5"]}


DATA
------------------------------------------------
{
  "_id" : 50,
  "_metadata" :
  {
    "etag" : "1591A6C3A20C4BC85C0498F7B1F4031F",
    "asof" : "000025653374F0C7"
  },
  "dname" : "MI6",
  "loc" : "LONDON",
  "emps" :
  [
    {
      "empno" : 1,
      "ename" : "M",
      "job" : "MANAGER",
      "hiredate" : "1940-01-01T00:00:00",
      "sal" : 42000,
      "comm" : 8000
    },
    {
      "empno" : 7,
      "ename" : "BOND",
      "job" : "AGENT",
      "mgr" : 1,
      "mgrname" : "M",
      "hiredate" : "1950-01-01T00:00:00",
      "sal" : 21001,
      "comm" : null,
      "tools" :
      [
        "Knife",
        "Garrote Watch",
        "Walther PPK",
        "Aston Martin DB5"
      ]
    }
  ],
  "secret" : true,
  "street" : "85 Albert Embankment"
}

9. Delete From Duality View

And now, let’s delete department 50 with all its employees to restore the original content of the dept and emp tables.

delete dept_dv v
 where v.data."_id".numberOnly() = 50;
commit;

select * from dept order by deptno;
select * from emp order by deptno, empno;

1 row deleted.


Commit complete.


    DEPTNO DNAME          LOC           EXT
---------- -------------- ------------- --------------------
        10 ACCOUNTING     NEW YORK
        20 RESEARCH       DALLAS
        30 SALES          CHICAGO
        40 OPERATIONS     BOSTON


     EMPNO ENAME      JOB              MGR HIREDATE          SAL       COMM     DEPTNO EXT
---------- ---------- --------- ---------- ---------- ---------- ---------- ---------- --------------------
      7782 CLARK      MANAGER         7839 1981-06-09       2450                    10
      7839 KING       PRESIDENT            1981-11-17       5000                    10
      7934 MILLER     CLERK           7782 1982-01-23       1300                    10
      7369 SMITH      CLERK           7902 1980-12-17        800                    20
      7566 JONES      MANAGER         7839 1981-04-02       2975                    20
      7788 SCOTT      ANALYST         7566 1987-04-19       3000                    20
      7876 ADAMS      CLERK           7788 1987-05-23       1100                    20
      7902 FORD       ANALYST         7566 1981-12-03       3000                    20
      7499 ALLEN      SALESMAN        7698 1981-02-20       1600        300         30
      7521 WARD       SALESMAN        7698 1981-02-22       1250        500         30
      7654 MARTIN     SALESMAN        7698 1981-09-28       1250       1400         30
      7698 BLAKE      MANAGER         7839 1981-05-01       2850                    30
      7844 TURNER     SALESMAN        7698 1981-09-08       1500          0         30
      7900 JAMES      CLERK           7698 1981-12-03        950                    30

14 rows selected.

Outlook

One thing is missing in version 0.9 of IslandSQL grammar. The support of PL/pgSQL in PostgreSQL statements create function, create procedure, create trigger and do. These statements can already be parsed but the PL/pgSQL code passed as string is not further analyzed. This will change in the next and final episode of this season.

The post IslandSQL Episode 9: GraphQL, JSON and Flexible Schemas With Duality Views appeared first on Philipp Salvisberg's Blog.

In the last episode, we looked at some features in PostgreSQL which I miss in the Oracle Database. The IslandSQL grammar now covers PL/SQL and the related DDL statements. The implementation was more complex than expected, mainly because of the incompatibilities between PostgreSQL and the Oracle Database. I will probably deal with this topic in a future blog post.

Now I’d like to talk about the new features in Oracle Database 23ai. Not about all features in the New Features Guide, but only about some changes since the release of 23.3, which was known as 23c. It’s hard to find changes from 23.3 to 23.4 in the documentation. So, I guess it’s worth a blog post. I focus on the features that are relevant to the IslandSQL grammar. In other words, the interesting ones from a developer’s perspective.

1. Vector Data Type

A vector is a number array for which you can optionally define the number of dimensions (size) and the data type (int8, float32 or float64) of the dimension values. This data type is the basis for the vector search functionality.

Here’s a slightly amended example from the documentation creating and populating a table with a vector data type.

drop table if exists galaxies purge;
create table galaxies (
   id        number             not null primary key,
   name      varchar2(10 char)  not null unique,
   embedding vector(5, int8)    not null, -- 5 dimensions, stored as int8
   doc       varchar2(120 char) not null
);

insert into galaxies 
   (id, name, embedding, doc)
values 
   (1, 'M31',     '[0,2,2,0,0]', 'Messier 31 is a barred spiral galaxy in the Andromeda constellation which has a lot of barred spiral galaxies.'),
   (2, 'M33',     '[0,0,1,0,0]', 'Messier 33 is a spiral galaxy in the Triangulum constellation.'),
   (3, 'M58',     '[1,1,1,0,0]', 'Messier 58 is an intermediate barred spiral galaxy in the Virgo constellation.'),
   (4, 'M63',     '[0,0,1,0,0]', 'Messier 63 is a spiral galaxy in the Canes Venatici constellation.'),
   (5, 'M77',     '[0,1,1,0,0]', 'Messier 77 is a barred spiral galaxy in the Cetus constellation.'),
   (6, 'M91',     '[0,1,1,0,0]', 'Messier 91 is a barred spiral galaxy in the Coma Berenices constellation.'),
   (7, 'M49',     '[0,0,0,1,1]', 'Messier 49 is a giant elliptical galaxy in the Virgo constellation.'),
   (8, 'M60',     '[0,0,0,0,1]', 'Messier 60 is an elliptical galaxy in the Virgo constellation.'),
   (9, 'NGC1073', '[0,1,1,0,0]', 'NGC 1073 is a barred spiral galaxy in Cetus constellation.');
commit;

The vector in this example has 5 dimensions with the following meaning:

1. Number of occurrences of intermediate in the doc column
2. Number of occurrences of barred in the doc column
3. Number of occurrences of spiral in the doc column
4. Number of occurrences of giant in the doc column
5. Number of occurrences of elliptical in the doc column

2. Vector Functions and PL/SQL Packages

The vector_distance function is a key functionality for similarity searches. Let’s say we want to see galaxies that are similar to NGC1073 (barred and spiral). The following query shows how vector functions can help to get this result:

with ngc1073 as (select vector('[0,1,1,0,0]', 5, int8) as query_vector)
select name, 
       round(vector_distance(embedding, query_vector, cosine), 3) as cosine_distance,
       round(vector_distance(embedding, query_vector, dot), 3) as inner_product,
       round(vector_distance(embedding, query_vector, euclidean), 3) as l2_distance,
       round(vector_distance(embedding, query_vector, euclidean_squared), 3) as l2_squared,
       round(vector_distance(embedding, query_vector, hamming), 3) as hamming_distance,
       round(vector_distance(embedding, query_vector, manhattan), 3) as l1_distance
  from galaxies, ngc1073
 order by cosine_distance;

NAME       COSINE_DISTANCE INNER_PRODUCT L2_DISTANCE L2_SQUARED HAMMING_DISTANCE L1_DISTANCE
---------- --------------- ------------- ----------- ---------- ---------------- -----------
M31                      0            -4       1.414          2                2           2
M77                      0            -2           0          0                0           0
M91                      0            -2           0          0                0           0
NGC1073                  0            -2           0          0                0           0
M58                   .184            -2           1          1                1           1
M63                   .293            -1           1          1                1           1
M33                   .293            -1           1          1                1           1
M60                      1             0       1.732          3                3           3
M49                      1             0           2          4                4           4

9 rows selected.

M77 and M91 have identical vectors and they are expected to be very similar. However, M31 is interesting. It has a similar shape to NGC1073 but the words barred and spiral appear twice. This is a good match only for some distance metrics.

A conventional query, e.g. based on Oracle Text might be good enough to find similar galaxies. However, if you have vectors with tons of dimensions then a vector similarity search might become appealing. Especially, since you can index vector columns to speed up your queries.

The following SQL functions are provided for the vector data type in the Oracle Database:

• vector_chunks
• vector or to_vector
• vector_dimension_count or vector_dims
• vector_dimension_format
• vector_distance or cosine_distance or inner_product or l1_distance or l2_distance
• vector_embedding
• vector_norm
• vector_serialize or from_vector

Furthermore, the following supplied PL/SQL packages provide additional functionality related to the vector data type:

• dbms_vector
• dbms_vector_chain

3. Shorthand Operators for Distances

For some vector_distance metrics shorthand operators (<=>, <->, <#>) are available. Here’s an example:

with ngc1073 as (select vector('[0,1,1,0,0]', 5, int8) as query_vector)
select name, 
       round(embedding <=> query_vector, 3) as cosine_distance,
       round(embedding <-> query_vector, 3) as l2_distance,
       round(embedding <#> query_vector, 3) as inner_product
  from galaxies, ngc1073
 order by cosine_distance;

NAME       COSINE_DISTANCE L2_DISTANCE INNER_PRODUCT
---------- --------------- ----------- -------------
M31                      0       1.414            -4
M77                      0           0            -2
M91                      0           0            -2
NGC1073                  0           0            -2
M58                   .184           1            -2
M63                   .293           1            -1
M33                   .293           1            -1
M60                      1       1.732             0
M49                      1           2             0

9 rows selected. 

4. Approximate Similarity Searches

You can imagine producing a top-N result of the previous queries using the row_limiting_clause. This clause was introduced in 12.1 to limit search results. Exactly to be precise. In 23.4 the clause was extended to support approximate similarity searches. The idea is to get a good enough result with a better performance when using vector indexes. Here’s an example:

select name, embedding
  from galaxies
 order by embedding <=> vector('[0,1,1,0,0]', 5, int8)
 fetch approx first 3 rows only
  with target accuracy 80 percent;

NAME       EMBEDDING           
---------- --------------------
M31        [0,2,2,0,0]         
M91        [0,1,1,0,0]         
M77        [0,1,1,0,0]         

5. Source and Destination Predicates in Graph Operator

The graph operator is a new 23ai feature. In 23.4 it got two additional predicates: source_predicate and destination_predicate. It allows us to test if a vertex is a source or a destination of an edge. The direction of the arrow so to speak. Here’s a formatted example from the documentation, based on this model.

select *
  from graph_table (students_graph
          match (p1 is person) -[e is friends]- (p2 is person)
          where p1.name = 'Mary'
          columns (
             e.friendship_id,
             e.meeting_date,
             case 
                when p1 is source of e then 
                   p1.name 
                else
                   p2.name 
             end as from_person,
             case 
                when p1 is destination of e then
                   p1.name
               else
                   p2.name 
             end as to_person
          )
       )
 order by friendship_id;

FRIENDSHIP_ID MEETING_DATE        FROM_PERSON TO_PERSON 
------------- ------------------- ----------- ----------
            1 19.09.2000 00:00:00 Mary        Alice     
            5 19.09.2000 00:00:00 Mary        John      
            7 10.07.2001 00:00:00 Bob         Mary 

6. Breaking Change for Inlined MLE Call Specification

In 23.3 the following code works.

create or replace function get42 return number is 
   mle language javascript q'[return 42;]';
/
select get42();

Function GET42 compiled


   GET42()
----------
        42

In 24.4 the same code produces this error:

Function GET42 compiled

LINE/COL  ERROR
--------- -------------------------------------------------------------
2/28      PLS-00881: missing closing delimiter 'q'[return' for MLE language code
Errors: check compiler log

The reason is a change in the syntax of the inlined MLE call specification. The JavaScript code cannot be passed as a string anymore. Instead, a new type of delimiter must be used as documented here. You can use almost any sequence of characters as a delimiter for the JavaScript code. The same character sequence must be used for the start and the end delimiter. Exceptions are the pairs (), [], {} and <>.

PostgreSQL dollar-quoted string constants are also valid delimiters. This works in 23.4:

create or replace function get42 return number is 
   mle language javascript $code$ return 42;$code$;
/

select get42();

Function GET42 compiled


   GET42()
----------
        42

Please note that the space after the first $code$ is required for the Oracle Database to recognize the end of the delimiter.

This change might simplify the implementation of additional MLE languages. Nevertheless, this is a breaking change that requires an amendment of the existing code base.

Outlook

In the next episode, the IslandSQL grammar will be extended to cover the missing statements with a query block. This means the

• create view,
• create materialized view and
• create table statement

for PostgreSQL and the Oracle Database. Only two episodes are left until the end of the first IslandSQL season.

The post IslandSQL Episode 8: What’s New in Oracle Database 23.4? appeared first on Philipp Salvisberg's Blog.