Arbori Archives - Philipp Salvisberg's Blog https://www.salvis.com/blog/tag/arbori/ Database-centric development Sun, 12 Nov 2023 12:57:40 +0000 en-US hourly 1 https://wordpress.org/?v=6.9.4 https://www.salvis.com/blog/wp-content/uploads/2014/04/favicon.png Arbori Archives - Philipp Salvisberg's Blog https://www.salvis.com/blog/tag/arbori/ 32 32 Quoted Identifiers #JoelKallmanDay https://www.salvis.com/blog/2022/10/11/quoted-identifiers-joelkallmanday/ Mon, 10 Oct 2022 22:42:05 +0000 https://www.salvis.com/blog/?p=11687 Background and TL;DR Connor McDonald wrote a blog post named Cleaner DDL than DBMS_METADATA. Back then he asked me if it would be possible to let the formatter remove unnecessary double quotes from quoted identifiers. Yes, of course. Actually, the current version of the PL/SQL & SQL Formatter Settings does exactly that. […]

The post Quoted Identifiers #JoelKallmanDay appeared first on Philipp Salvisberg's Blog.

]]> Background and TL;DR

Connor McDonald wrote a blog post named Cleaner DDL than DBMS_METADATA. Back then he asked me if it would be possible to let the formatter remove unnecessary double quotes from quoted identifiers. Yes, of course. Actually, the current version of the PL/SQL & SQL Formatter Settings does exactly that. And no, you cannot do that with dbms_metadata in the Oracle Database versions 19c and 21c. Read on if you are interested in the details.

The Problem with DBMS_METADATA …

When you execute a DDL against the Oracle Database the database stores some metadata in the data dictionary. The DDL statement itself is not stored, at least not completely. This is one reason why it is a good idea to store DDL statements in files and manage these files within a version control system. Nevertheless, the Oracle Database provides an API – dbms_metadata – to reconstruct a DDL based on the information available in the data dictionary.

Let’s create a view based on the famous dept and emp tables:

1) Initial DDL
create or replace view deptsal as
   select d.deptno,
          d.dname,
          nvl(sum(e.sal), 0) as sum_sal,
          nvl(count(e.empno), 0) as num_emps,
          nvl(round(avg(e.sal), 2), 0) as avg_sal
     from dept d
     left join emp e
       on e.deptno = d.deptno
    group by d.deptno, d.dname;

and retrieve the DDL like this:

2) Get DDL
select dbms_metadata.get_ddl('VIEW', 'DEPTSAL', user) from dual;

to produce this DDL:

3) DDL by dbms_metadata
CREATE OR REPLACE FORCE EDITIONABLE VIEW "REDSTACK"."DEPTSAL" ("DEPTNO", "DNAME", "SUM_SAL", "NUM_EMPS", "AVG_SAL") DEFAULT COLLATION "USING_NLS_COMP"  AS 
  select d.deptno,
          d.dname,
          nvl(sum(e.sal), 0) as sum_sal,
          nvl(count(e.empno), 0) as num_emps,
          nvl(round(avg(e.sal), 2), 0) as avg_sal
     from dept d
     left join emp e
       on e.deptno = d.deptno
    group by d.deptno, d.dname

We see that the subquery part of the view has been preserved, except the first line which has a different indentation. Actually, the indentation of the first line is not stored in the data dictionary (see user_views.text). The two spaces are produced by the default pretty option of dbms_metadata. So far, so good.

In many cases, the Oracle Data Dictionary explicitly stores default values. For instance, Y for editionable or USING_NLS_COMP for default_collation. This fact alone makes it impossible to reconstruct the original DDL reliably. The database simply does not know whether an optional clause such as editionable or default collation has been specified or omitted. Moreover, some optional DDL clauses such as or replace or force are simply not represented in the data dictionary.

… Especially with Quoted Identifiers

And last but not least, identifiers such as columns names, table names or view names are stored without double quotes. Therefore, the database knows nothing about the use of double quotes in the original DDL. However, the database knows exactly when double quotes are required. As a result, dbms_metadata could emit only necessary double quotes. This would result in a more readable DDL and would probably also be more similar to the original DDL.

The reality is that code generators such as dbms_metadata often use double quotes for all identifiers. It’s simply easier for them, because this way the generated code works for all kind of strange identifiers.

However, using quoted identifiers is a bad practice. It is, in fact, a very bad practice when they are used unnecessarily.

Shaping the DDL

So what can we do? We can configure dbms_metadata to produce a DDL which is more similar to our original one. In this case we can change the following:

  • remove the schema of the view (owner)
  • remove the force keyword
  • remove the default collation clause
  • add the missing SQL terminator (;)

This query

4) Get customized DDL
with
   function view_ddl(in_name in varchar2) return clob is
      l_main_handle   integer;
      l_modify_handle integer;
      l_ddl_handle    integer;
      l_ddl           clob;
   begin
      -- initialize dbms_metadata for view based on current schema
      l_main_handle   := sys.dbms_metadata.open('VIEW');
      sys.dbms_metadata.set_filter(l_main_handle, 'SCHEMA', user);
      sys.dbms_metadata.set_filter(l_main_handle, 'NAME', in_name);
      -- remove schema name from input structure
      l_modify_handle := sys.dbms_metadata.add_transform(l_main_handle, 'MODIFY');
      sys.dbms_metadata.set_remap_param(l_modify_handle, 'REMAP_SCHEMA', user, null);
      -- non-default transformations to improve DDL
      l_ddl_handle    := sys.dbms_metadata.add_transform(l_main_handle, 'DDL');
      sys.dbms_metadata.set_transform_param(l_ddl_handle, 'FORCE', false);
      sys.dbms_metadata.set_transform_param(l_ddl_handle, 'COLLATION_CLAUSE', 'NO_NLS');
      sys.dbms_metadata.set_transform_param(l_ddl_handle, 'SQLTERMINATOR', true);
      -- get DDL
      l_ddl           := sys.dbms_metadata.fetch_clob(l_main_handle);
      -- free sys.dbms_metadata resources
      sys.dbms_metadata.close(l_main_handle);
      -- return result
      return l_ddl;
   end view_ddl;
select view_ddl('DEPTSAL')
  from dual
/

produces this result:

5) Customized DDL via dbms_metadata
CREATE OR REPLACE EDITIONABLE VIEW "DEPTSAL" ("DEPTNO", "DNAME", "SUM_SAL", "NUM_EMPS", "AVG_SAL") AS 
  select d.deptno,
          d.dname,
          nvl(sum(e.sal), 0) as sum_sal,
          nvl(count(e.empno), 0) as num_emps,
          nvl(round(avg(e.sal), 2), 0) as avg_sal
     from dept d
     left join emp e
       on e.deptno = d.deptno
   group by d.deptno, d.dname;

This looks better. However, I would like to configure dbms_metadata to omit the default editionable clause. Furthermore, I do not like the column alias list, which is unnecessary in this case. And of course, I’d like to suppress unnecessary double quotes around identifiers. Is that possible with dbms_metadata?

Shaping the DDL from (S)XML

Well, we can try. The dbms_metadata API is very extensive. Besides other things, it can also represent metadata as an XML document. There are two formats.

  • XML – An extensive XML containing internals such as object number, owner number, creation date, etc.
  • SXML – A simple and terse XML that contains everything you need to produce a DDL. The SXML format is therefore very well suited for schema comparison.

It’s possible to produce a DDL from both formats. We can also change the XML beforehand.

Let’s look at both variants in the next two subchapters.

Important: I consider the changes to the XML document and configuration of dbms_metadata in the following subchapters as experimental. The purpose is to show what is doable. They are not good examples of how it should be done. Even though the unnecessary list of column aliases annoys me, I would leave them as they are. I also think that overriding the default VERSION is a very bad idea in the long run.

Convert XML to DDL

6) Get customized DDL from XML
with
   function view_ddl(in_name in varchar2) return clob is
      l_xml         xmltype;
      l_main_handle integer;
      l_ddl_handle  integer;
      l_ddl         clob;
   begin
      -- create XML document and remove unwanted nodes
      l_xml         := xmltype(sys.dbms_metadata.get_xml('VIEW', in_name, user));
      l_xml         := l_xml.deletexml('/ROWSET/ROW/VIEW_T/SCHEMA_OBJ/OWNER_NAME');
      l_xml         := l_xml.deletexml('/ROWSET/ROW/VIEW_T/COL_LIST');
      -- initialize dbms_metadata for view based on XML input
      l_main_handle := sys.dbms_metadata.openw('VIEW');
      -- non-default transformations to improve DDL
      l_ddl_handle  := sys.dbms_metadata.add_transform(l_main_handle, 'DDL');
      sys.dbms_metadata.set_transform_param(l_ddl_handle, 'FORCE', false);
      sys.dbms_metadata.set_transform_param(l_ddl_handle, 'COLLATION_CLAUSE', 'NO_NLS');
      sys.dbms_metadata.set_transform_param(l_ddl_handle, 'SQLTERMINATOR', true);
      sys.dbms_metadata.set_transform_param(l_ddl_handle, 'VERSION', 1120000000);
      -- get DDL
      sys.dbms_lob.createtemporary(l_ddl, false, sys.dbms_lob.session);
      sys.dbms_metadata.convert(l_main_handle, l_xml, l_ddl);
      -- free dbms_metadata resources
      sys.dbms_metadata.close(l_main_handle);
      -- return result
      return l_ddl;
   end view_ddl;
select xmlserialize(document xmltype(sys.dbms_metadata.get_xml('VIEW', 'DEPTSAL', user))
          as clob indent size = 4)
  from dual
union all
select view_ddl('DEPTSAL')
  from dual
/

The query produces the following two rows (CLOBs):

7) XML input
<?xml version="1.0"?>
<ROWSET>
    <ROW>
        <VIEW_T>
            <VERS_MAJOR>1</VERS_MAJOR>
            <VERS_MINOR>4 </VERS_MINOR>
            <OBJ_NUM>232322</OBJ_NUM>
            <SCHEMA_OBJ>
                <OBJ_NUM>232322</OBJ_NUM>
                <OWNER_NUM>501</OWNER_NUM>
                <OWNER_NAME>REDSTACK</OWNER_NAME>
                <NAME>DEPTSAL</NAME>
                <NAMESPACE>1</NAMESPACE>
                <TYPE_NUM>4</TYPE_NUM>
                <TYPE_NAME>VIEW</TYPE_NAME>
                <CTIME>2022-10-04 12:11:25</CTIME>
                <MTIME>2022-10-04 12:11:25</MTIME>
                <STIME>2022-10-04 12:11:25</STIME>
                <STATUS>1</STATUS>
                <FLAGS>0</FLAGS>
                <FLAGS2>0</FLAGS2>
                <SPARE1>6</SPARE1>
                <SPARE2>65535</SPARE2>
                <SPARE3>501</SPARE3>
                <OWNER_NAME2>REDSTACK</OWNER_NAME2>
                <SIGNATURE>76DCDE35671FAA6AF576D6A6B4D97D48</SIGNATURE>
                <SPARE7>134233583</SPARE7>
                <SPARE8>0</SPARE8>
                <SPARE9>0</SPARE9>
                <DFLCOLLNAME>USING_NLS_COMP</DFLCOLLNAME>
            </SCHEMA_OBJ>
            <AUDIT_VAL>--------------------------------------</AUDIT_VAL>
            <COLS>5</COLS>
            <INTCOLS>5</INTCOLS>
            <PROPERTY>0</PROPERTY>
            <PROPERTY2>0</PROPERTY2>
            <FLAGS>0</FLAGS>
            <TEXTLENGTH>270</TEXTLENGTH>
            <TEXT>select d.deptno,
          d.dname,
          nvl(sum(e.sal), 0) as sum_sal,
          nvl(count(e.empno), 0) as num_emps,
          nvl(round(avg(e.sal), 2), 0) as avg_sal
     from dept d
     left join emp e
       on e.deptno = d.deptno
   group by d.deptno, d.dname</TEXT>
            <COL_LIST>
                <COL_LIST_ITEM>
                    <OBJ_NUM>232322</OBJ_NUM>
                    <COL_NUM>1</COL_NUM>
                    <INTCOL_NUM>1</INTCOL_NUM>
                    <SEGCOL_NUM>1</SEGCOL_NUM>
                    <PROPERTY>0</PROPERTY>
                    <PROPERTY2>0</PROPERTY2>
                    <NAME>DEPTNO</NAME>
                    <TYPE_NUM>2</TYPE_NUM>
                </COL_LIST_ITEM>
                <COL_LIST_ITEM>
                    <OBJ_NUM>232322</OBJ_NUM>
                    <COL_NUM>2</COL_NUM>
                    <INTCOL_NUM>2</INTCOL_NUM>
                    <SEGCOL_NUM>2</SEGCOL_NUM>
                    <PROPERTY>0</PROPERTY>
                    <PROPERTY2>0</PROPERTY2>
                    <NAME>DNAME</NAME>
                    <TYPE_NUM>1</TYPE_NUM>
                </COL_LIST_ITEM>
                <COL_LIST_ITEM>
                    <OBJ_NUM>232322</OBJ_NUM>
                    <COL_NUM>3</COL_NUM>
                    <INTCOL_NUM>3</INTCOL_NUM>
                    <SEGCOL_NUM>3</SEGCOL_NUM>
                    <PROPERTY>14336</PROPERTY>
                    <PROPERTY2>0</PROPERTY2>
                    <NAME>SUM_SAL</NAME>
                    <TYPE_NUM>2</TYPE_NUM>
                </COL_LIST_ITEM>
                <COL_LIST_ITEM>
                    <OBJ_NUM>232322</OBJ_NUM>
                    <COL_NUM>4</COL_NUM>
                    <INTCOL_NUM>4</INTCOL_NUM>
                    <SEGCOL_NUM>4</SEGCOL_NUM>
                    <PROPERTY>14336</PROPERTY>
                    <PROPERTY2>0</PROPERTY2>
                    <NAME>NUM_EMPS</NAME>
                    <TYPE_NUM>2</TYPE_NUM>
                </COL_LIST_ITEM>
                <COL_LIST_ITEM>
                    <OBJ_NUM>232322</OBJ_NUM>
                    <COL_NUM>5</COL_NUM>
                    <INTCOL_NUM>5</INTCOL_NUM>
                    <SEGCOL_NUM>5</SEGCOL_NUM>
                    <PROPERTY>14336</PROPERTY>
                    <PROPERTY2>0</PROPERTY2>
                    <NAME>AVG_SAL</NAME>
                    <TYPE_NUM>2</TYPE_NUM>
                </COL_LIST_ITEM>
            </COL_LIST>
            <COL_LIST2/>
            <CON1_LIST/>
            <CON2_LIST/>
        </VIEW_T>
    </ROW>
</ROWSET>
8) Customized DDL from XML
    CREATE OR REPLACE VIEW "DEPTSAL" () AS 
  select d.deptno,
          d.dname,
          nvl(sum(e.sal), 0) as sum_sal,
          nvl(count(e.empno), 0) as num_emps,
          nvl(round(avg(e.sal), 2), 0) as avg_sal
     from dept d
     left join emp e
       on e.deptno = d.deptno
    group by d.deptno, d.dname;

We removed the OWNER_NAME node (on line 11) from the XML document. As a result, the schema was removed in the DDL. The result is the same as with the REMAP_SCHEMA transformation. Perfect.

We also removed the COL_LIST node (lines 48-99) from the XML document. However, the result in the DDL regarding the column alias list does not look good. The columns are gone, but the surrounding parentheses survived, which makes the DDL invalid. IMO this is a bug in the $ORACLE_HOME/rdbms/xml/xsl/kuview.xsl script. It’s handled correctly in the SXML script as we will see later. However, we can fix that by calling replace(..., '" () AS', '" AS') . Please note that a complete solution should do some checks to ensure that the COL_LIST is really not required.

When you look at line 12 in the XML document (<NAME>DEPTSAL</NAME>), you see that the view name does not contain double quotes. This is a strong indicator, that there is no way to remove the double quotes by manipulating the input XML document. In fact, the double quotes are hard-coded in all XSLT scripts. No way to override this behavior via dbms_metadata.

Furthermore, you do not find a node named like EDITIONABLEwith a value Y as in all_objects. Why? Because this information is stored in the node FLAGS. 0 means editionable and 1048576 means noneditionable. To be precise 1048576 represents bit number 21. If this bit is set then the view is noneditionable. You find the proof for this statement in the dba_objects view, where the expression for the editionable column looks like this:

9) Editionable column in dba_objects (formatted)
case
   when o.type# in (
      4, 5, 7, 8, 9, 11, 12, 13, 14, 22, 87, 114
   )
   then
      decode(
         bitand(o.flags, 1048576),
            0,       'Y', 
            1048576, 'N', 
                     'Y'
      )
   else
      null
end

The $ORACLE_HOME/rdbms/xml/xsl/kucommon.xslscript (see template Editionable) is evaluating this flag and either emitting a EDITIONABLE or NONEDITIONABLE text. These keywords were introduced in version 12.1. Since dbms_metadata can produce version-specific DDL, we set the version to 11.2 to suppress EDITIONABLE in the resulting DDL.

Convert SXML to DDL

10) Get customized DDL from SXML
with
   function view_ddl(in_name in varchar2) return clob is
      l_sxml        xmltype;
      l_main_handle integer;
      l_ddl_handle  integer;
      l_ddl         clob;
   begin
      -- create SXML document and remove unwanted nodes
      l_sxml        := xmltype(sys.dbms_metadata.get_sxml('VIEW', in_name, user));
      l_sxml        := l_sxml.deletexml('/VIEW/SCHEMA', 'xmlns=http://xmlns.oracle.com/ku');
      l_sxml        := l_sxml.deletexml('/VIEW/COL_LIST', 'xmlns=http://xmlns.oracle.com/ku');
      -- initialize dbms_metadata for view based on SXML input
      l_main_handle := sys.dbms_metadata.openw('VIEW');
      -- non-default transformations to improve DDL
      l_ddl_handle  := sys.dbms_metadata.add_transform(l_main_handle, 'SXMLDDL');
      sys.dbms_metadata.set_transform_param(l_ddl_handle, 'FORCE', false);
      sys.dbms_metadata.set_transform_param(l_ddl_handle, 'COLLATION_CLAUSE', 'NO_NLS');
      sys.dbms_metadata.set_transform_param(l_ddl_handle, 'SQLTERMINATOR', true);
      sys.dbms_metadata.set_transform_param(l_ddl_handle, 'VERSION', 1120000000);
      -- get DDL
      sys.dbms_lob.createtemporary(l_ddl, false, sys.dbms_lob.session);
      sys.dbms_metadata.convert(l_main_handle, l_sxml, l_ddl);
      -- free dbms_metadata resources
      sys.dbms_metadata.close(l_main_handle);
      -- return result
      return l_ddl;
   end view_ddl;
select xmlserialize(document xmltype(sys.dbms_metadata.get_sxml('VIEW', 'DEPTSAL', user))
          as clob indent size = 4)
  from dual
union all
select view_ddl('DEPTSAL')
  from dual
/
11) SXML input
<VIEW xmlns="http://xmlns.oracle.com/ku" version="1.0">
    <SCHEMA>REDSTACK</SCHEMA>
    <NAME>DEPTSAL</NAME>
    <DEFAULT_COLLATION>USING_NLS_COMP</DEFAULT_COLLATION>
    <COL_LIST>
        <COL_LIST_ITEM>
            <NAME>DEPTNO</NAME>
        </COL_LIST_ITEM>
        <COL_LIST_ITEM>
            <NAME>DNAME</NAME>
        </COL_LIST_ITEM>
        <COL_LIST_ITEM>
            <NAME>SUM_SAL</NAME>
        </COL_LIST_ITEM>
        <COL_LIST_ITEM>
            <NAME>NUM_EMPS</NAME>
        </COL_LIST_ITEM>
        <COL_LIST_ITEM>
            <NAME>AVG_SAL</NAME>
        </COL_LIST_ITEM>
    </COL_LIST>
    <SUBQUERY>select d.deptno,
          d.dname,
          nvl(sum(e.sal), 0) as sum_sal,
          nvl(count(e.empno), 0) as num_emps,
          nvl(round(avg(e.sal), 2), 0) as avg_sal
     from dept d
     left join emp e
       on e.deptno = d.deptno
   group by d.deptno, d.dname</SUBQUERY>
</VIEW>
12) Customized DDL from SXML
CREATE OR REPLACE VIEW ""."DEPTSAL" 
  AS 
  select d.deptno,
          d.dname,
          nvl(sum(e.sal), 0) as sum_sal,
          nvl(count(e.empno), 0) as num_emps,
          nvl(round(avg(e.sal), 2), 0) as avg_sal
     from dept d
     left join emp e
       on e.deptno = d.deptno
    group by d.deptno, d.dname;

The SXML document is smaller. It contains just the nodes to produce a DDL. That makes it easier to read.

We removed the SCHEMA node (on line 2) from the SXML document. As a result, the schema was removed in the DDL. But not completely. Two double quotes and one dot survived, which makes the DDL invalid. IMO this is a bug in the $ORACLE_HOME/rdbms/xml/xsl/kusviewd.xsl script. It’s handled correctly in the XML script. We could fix that with a replace(..., 'VIEW ""."', 'VIEW "') call. As long as the search term is not ambiguous, everything should be fine.

We also removed the COL_LIST node (lines 5-21) from the SXML document. In this case, the column alias list is completely removed from the DDL. Including the parentheses. Nice.

Maybe you wonder how editionable is represented in the SXML document. – With a NONEDITIONABLE node if the view is noneditionable.

How Can We Work Around the Limitations?

We’ve seen the limitations of the current dbms_metadata API and the necessity to use string manipulation functions to fix invalid DDL.

There is no way to remove double quotes from quoted identifiers with dbms_metadata. However, as Connor McDonald demonstrated in his blog post we can remove them with some string acrobatics. Why not use a simple replace call? Because there are some rules to follow. A globally applied replace(..., '"', null) call would produce invalid code in many real-life scenarios. We need a more robust solution.

Applying the rules in a code formatter can be such a robust solution.

Rules for Safely Removing Double Quotes from Quoted Identifiers

What are the rules to follow?

1. Is a SQL or PL/SQL Identifier

You have to make sure that the double quotes surround a SQL or PL/SQL identifier. Sounds logical. However, it is not that simple. Here are some examples:

13) PL/SQL comment
create or replace procedure plsql_comment is
begin
   -- "NOT_AN_IDENTIFIER"
   /*
      "NOT_AN_IDENTIFIER"
   */
   null;
end;
/
14) PL/SQL string
create or replace procedure plsql_string is
   l_string1 varchar2(100 char);
   l_string2 varchar2(100 char);
begin
   l_string1 := '"NOT_AN_IDENTIFIER"';
   l_string2 := q'[
                   "NOT_AN_IDENTIFIER"
                ]';
end;
/
15) PL/SQL conditional compilation text
create or replace procedure plsql_conditional_compilation_text is
begin
   $if false $then
      Conditional compilation blocks can contain any text.
      It does not need to be valid PL/SQL.
      "NOT_AN_IDENTIFIER"
      FTLDB and tePLSQL use this construct to store code templates in such blocks.
   $end
   null;
end;
/
16) Java string
create or replace and resolve java source named "JavaString" as
public class JavaString {
  public static String hello() {
     return "NOT_AN_IDENTIFIER";
  }
}
/

You can solve the first three examples easily with a lexer. A lexer groups a stream of characters. Such a group of characters is called a lexer token. A lexer token knows the start and end position in a source text and has a type. The lexer in SQL Developer and SQLcl produces the following types of tokens:

  • COMMENT (/* ... */)
  • LINE_COMMENT (-- ...)
  • QUOTED_STRING ('string' or q'[string]')
  • DQUOTED_STRING ("string" )
  • WS (space, tab, new line, carriage return)
  • DIGITS (0123456789 plus some special cases)
  • OPERATION (e.g. ()[]^-|!*+.><=,;:%@?/~)
  • IDENTIFIER (words)
  • MACRO_SKIP (conditional compilation tokens such as $if, $then, etc.)

We can simply focus on tokens of the type DQUOTED_STRING and ignore tokens that are within conditional compilation tokens $ifand $end.

To find out if a DQUOTED_STRING is part of a Java stored procedure is more difficult. Luckily SQL Developer’s parser cannot deal with Java stored procedures and produces a parse error. As a result, we just have to keep the code “as is” in such cases.

2. Consists of Valid Characters

According to the PL/SQL Language Reference a nonquoted identifier must comply with the following rules:

An ordinary user-defined identifier:

  • Begins with a letter
  • Can include letters, digits, and these symbols:
    • Dollar sign ($)
    • Number sign (#)
    • Underscore (_)

What is a valid letter in this context? The SQL Language Reference defines a letter as an “alphabetic character from your database character set”. Here are some examples of valid letters and therefore valid PL/SQL variable names or SQL column names:

  • Latin letters (AaBbCcDdEeFfGgHhIiJjKkLlMmNnOoPpQqRrSsTtUuVvWwXxYyZz)
  • Umlauts (ÄäËëÏïÖöÜüŸÿ)
  • German Esszett (ẞß), please note that the Oracle Database does not convert the case of an Esszett, because the uppercase Esszett exists offically only since 2017-03-29
  • C cedilla (Çç)
  • Grave accented letters (ÀàÈèÌìÒòÙù)
  • Acute accented letters (ÁáĆćÉéÍíÓóÚúÝý)
  • Circumflex accented letters (ÂâÊêÎîÔôÛû)
  • Tilde accented letters (ÃãÑñÕõ)
  • Caron accented letters (ǍǎB̌b̌ČčĚěF̌f̌ǦǧȞȟǏǐJ̌ǰǨǩM̌m̌ŇňǑǒP̌p̌Q̌q̌ŘřŠšǓǔV̌v̌W̌w̌X̌x̌Y̌y̌ŽžǮǯ)
  • Ring accented letters (ÅåŮů)
  • Greek letters (ΑαΒβΓγΔδΕεΖζΗηΘθΙιΚκΛλΜμΝνΞξΟοΠπΡρΣσΤτΥυΦφΧχΨψΩω)
  • Common Cyrillic letters (АаБбВвГгДдЕеЁёЖжЗзИиЙйКкЛлМмНнОоПпРрСсТтУуФфХхЦцЧчШшЩщЪъЫыЬьЭэЮюЯя)
  • Hiragana letters (ぁあぃいぅうぇえぉおかがきぎくぐけげこごさざしじすずせぜそぞただちぢっつづてでとどなにぬねのはばぱひびぴふぶぷへべぺほぼぽまみむめもゃやゅゆょよらりるれろゎわゐゑをんゔゕゖゝゞゟ)

The Oracle Database throws an ORA-00911: invalid character when this rule is violated.

Cause: The identifier name started with an ASCII character other than a letter or a number . After the first character of the identifier name, ASCII characters are allowed including “$”, “#” and “_”. Identifiers enclosed in double quotation marks may contain any character other than a double quotation. Alternate quotation marks (q’#…#’) cannot use spaces, tabs, or carriage returns as delimiters. For all other contexts, consult the SQL Language Reference Manual.

The cause for this error message seems to be outdated, inaccurate and wrong. Firstly, it limits letters to those contained in an ASCII character set. This limitation is not generally valid anymore. Secondly, it claims that an identifier can start with a number, which is simply wrong. Thirdly, ASCII characters and letters are used as synonyms, which is misleading.

However, there are still cases where an identifier is limited to ASCII characters or single byte characters. For example, a database name or a database link name. In the projects I know, the reduction of letters to A-Z for identifiers is not a problem. The use of accented letters in identifiers are typically oversights. Therefore, I recommend limiting the range of letters in identifiers to A-Z .

Checking this rule is quite simple. We just have to make sure that the quoted identifier matches this regular expression: ^"[A-Z][A-Z0-9_$#]*"$. This works with any regular expression engine, unlike ^"[[:alpha:]][[:alpha:]0-9_$#]*"$.

3. Is Not a Reserved Word

According to the PL/SQL Language Reference and the SQL Language Reference a nonquoted identifier must not be a reserved word.

If you are working with 3rd party parsers, the list of reserved words might not match the ones defined for the Oracle Database. In my case, I also want to consider the reserved words defined by db* CODECOP. I’m using the following query to create a JSON array with currently 260 keywords:

17) JSON array of keywords
select json_arrayagg(keyword order by keyword) as keywords
  from (
          -- reserved keywords in Oracle database 21.3.0.0.0 (ATP)
          select keyword
            from v$reserved_words
           where (reserved = 'Y' or res_type = 'Y' or res_attr = 'Y' or res_semi = 'Y')
             and keyword is not null
             and regexp_like(keyword, '^[A-Z][A-Z0-9_$#]*$') -- valid nonquoted identifier
          union
          -- reserved keywords in db* CODECOP's PL/SQL parser 4.2.0
          select keyword
            from json_table(
                    '["AFTER","ALL","ALLOW","ALTER","ANALYTIC","AND","ANYSCHEMA","AS","ASC","ASSOCIATE","AUTHID","AUTOMATIC",
                      "AUTONOMOUS_TRANSACTION","BEFORE","BEGIN","BETWEEN","BULK","BY","BYTE","CANONICAL","CASE",
                      "CASE-SENSITIVE","CHECK","CLUSTER","COMPOUND","CONNECT","CONNECT_BY_ROOT","CONSTANT","CONSTRAINT",
                      "CONSTRUCTOR","CORRUPT_XID","CORRUPT_XID_ALL","CREATE","CROSSEDITION","CURRENT","CUSTOMDATUM",
                      "CYCLE","DB_ROLE_CHANGE","DECLARE","DECREMENT","DEFAULTS","DEFINE","DEFINER","DETERMINISTIC",
                      "DIMENSION","DISALLOW","DISASSOCIATE","DISTINCT","DROP","EACH","EDITIONING","ELSE","ELSIF",
                      "END","EVALNAME","EXCEPTION","EXCEPTION_INIT","EXCEPTIONS","EXCLUSIVE","EXTERNAL","FETCH",
                      "FOLLOWING","FOLLOWS","FOR","FORALL","FROM","GOTO","GRANT","GROUP","HAVING","HIDE","HIER_ANCESTOR",
                      "HIER_LAG","HIER_LEAD","HIER_PARENT","IF","IGNORE","IMMUTABLE","IN","INCREMENT","INDEX","INDICATOR",
                      "INDICES","INITIALLY","INLINE","INSERT","INSTEAD","INTERSECT","INTO","INVISIBLE","IS","ISOLATION",
                      "JAVA","JSON_EXISTS","JSON_TABLE","LATERAL","LIBRARY","LIKE","LIKE2","LIKE4","LIKEC","LOCK","LOGON",
                      "MAXVALUE","MEASURES","MERGE","MINUS","MINVALUE","MULTISET","MUTABLE","NAN","NAV","NCHAR_CS","NOCOPY",
                      "NOCYCLE","NONSCHEMA","NORELY","NOT","NOVALIDATE","NOWAIT","OF","ON","ONLY","OPTION","OR","ORADATA",
                      "ORDER","ORDINALITY","OVER","OVERRIDING","PARALLEL_ENABLE","PARTITION","PASSING","PAST","PIPELINED",
                      "PIVOT","PRAGMA","PRECEDES","PRECEDING","PRESENT","PRIOR","PROCEDURE","REFERENCES","REFERENCING",
                      "REJECT","RELY","REPEAT","RESPECT","RESTRICT_REFERENCES","RESULT_CACHE","RETURNING","REVOKE",
                      "SELECT","SEQUENTIAL","SERIALIZABLE","SERIALLY_REUSABLE","SERVERERROR","SETS","SHARE","SIBLINGS",
                      "SINGLE","SOME","SQL_MACRO","SQLDATA","STANDALONE","START","SUBMULTISET","SUBPARTITION",
                      "SUPPRESSES_WARNING_6009","THE","THEN","TO","TRIGGER","UDF","UNBOUNDED","UNDER","UNION",
                      "UNIQUE","UNLIMITED","UNPIVOT","UNTIL","UPDATE","UPSERT","USING","VALUES","VARRAY","VARYING",
                      "VIEW","WHEN","WHERE","WHILE","WINDOW","WITH","XMLATTRIBUTES","XMLEXISTS","XMLFOREST",
                      "XMLNAMESPACES","XMLQUERY","XMLROOT","XMLSCHEMA","XMLSERIALIZE","XMLTABLE"]',
                    '$[*]' columns (keyword path '$')
                 )
       );

The result can be used to populate a HashSet. This allows you to check very efficiently whether an identifier is a keyword.

Of course, such a global list of keywords is a simplification. In reality, the restrictions are context-specific. However, I consider the use of keywords for identifiers in any context a bad practice. Therefore, I can live with some unnecessarily quoted identifiers.

4. Is in Upper Case

This means that the following condition must be true: quoted_identifier = upper(quoted_identifier).

This does not necessarily mean that the identifier is case-insensitive as the following examples show:

18) JSON dot notation is case-sensitive
set pagesize 100
column key format A3
column value format A5
set null "(-)"
 
-- OK, KEY/VALUE are clearly case-sensitive
select j.pair."KEY", j.pair."VALUE"
  from json_table('[{KEY:1, VALUE:"One"},{KEY:2, VALUE:"Two"}]',
          '$[*]' columns (pair varchar2(100) format json path '$')) j;
 
KEY VALUE
--- -----
1   One  
2   Two  
 
-- OK, KEY/VALUE are case-sensitive, but you have to "know" that
select j.pair.KEY, j.pair.VALUE
  from json_table('[{KEY:1, VALUE:"One"},{KEY:2, VALUE:"Two"}]',
          '$[*]' columns (pair varchar2(100) format json path '$')) j;
 
KEY VALUE
--- -----
1   One  
2   Two  
          
-- Oups, no error, but the result is wrong (all NULLs)
-- This is why you should not let the formatter change the case of your identifiers!
select j.pair.key, j.pair.value
  from json_table('[{KEY:1, VALUE:"One"},{KEY:2, VALUE:"Two"}]',
          '$[*]' columns (pair varchar2(100) format json path '$')) j;
 
KEY VALUE
--- -----
(-) (-)  
(-) (-)

You can check this rule in combination with the previous rule by using a case-insensitive regular expression, which is the default.

5. Is Not Part of a Code Section for Which the Formatter Is Disabled

When you use a formatter there are some code sections where you do not want the formatter to change it. Therefore we want to honor the marker comments that disable and enable the formatter.

Here is an example:

19) Disable formatter – input
create or replace procedure disable_enable_formatter is
   l_dummy sys.dual.dummy%type;
begin
   -- @formatter:off
   select decode(dummy, 'X', 1 
                      , 'Y', 2
                      , 'Z', 3
                           , 0) "DECODE_RESULT" /* @formatter:on */
     into "L_DUMMY"
     from "SYS"."DUAL";
 
   select "DUMMY" -- noformat start
     into "L_DUMMY"
     from "SYS"."DUAL" -- noformat end
    where "DUMMY" is not null;
end;
/

After calling the formatter we expect the following output (when changing the identifier case to lower is enabled):

20) Disable formatter – result
create or replace procedure disable_enable_formatter is
   l_dummy sys.dual.dummy%type;
begin
   -- @formatter:off
   select decode(dummy, 'X', 1 
                      , 'Y', 2
                      , 'Z', 3
                           , 0) "DECODE_RESULT" /* @formatter:on */
     into l_dummy
     from sys.dual;
 
   select dummy -- noformat start
     into "L_DUMMY"
     from "SYS"."DUAL" -- noformat end
    where dummy is not null;
end;
/

To check this we can reuse the approach for quoted identifiers in conditional compilation text.

Removing Double Quotes from Quoted Identifiers with Arbori

As mentioned at the beginning of the post, the current version of the PL/SQL & SQL Formatter Settings can safely remove double quotes from PL/SQL and SQL code.

Here are simplified formatter settings which can be imported into SQL Developer 22.2.1. The formatter with these settings only removes the double quotes from identifiers in a safe way and leaves your code “as is”. You can download these settings from this Gist.

21) Advanced Settings
<options>
    <adjustCaseOnly>false</adjustCaseOnly>
    <singleLineComments>oracle.dbtools.app.Format.InlineComments.CommentsUnchanged</singleLineComments>
    <maxCharLineSize>120000</maxCharLineSize>
    <idCase>oracle.dbtools.app.Format.Case.NoCaseChange</idCase>
    <kwCase>oracle.dbtools.app.Format.Case.lower</kwCase>
    <formatWhenSyntaxError>false</formatWhenSyntaxError>
</options>

Firstly, the option adjustCaseOnly ensures that the Arbori program is fully applied.

Secondly, the option singleLineComments ensures that the whitespace before single-line comments are kept “as is”.

Thirdly, the maxCharLineSize ensures that no line breaks are added. The value of 120000 seems to be ridiculously high. However, I’ve seen single lines of around a hundred thousand characters in the wild.

Fourthly, the option idCase ensures that the case of nonquoted identifiers is not changed. This is important for JSON dot notation.

Fifthly, the option kwCase ensures that the case of keywords is also kept “as is”.

And Finally, the option formatWhenSyntaxError ensures that the formatter does not change code that the formatter does not understand. This is important to keep Java strings intact.

The value of all other options is irrelevant for this Arbori program.

22) Custom Format
-- --------------------------------------------------------------------------------------------------------------------
-- Minimal Arbori program (expected by the formatter, also expected: "order_by_clause___0").
-- --------------------------------------------------------------------------------------------------------------------
include "std.arbori"
dummy: :indentConditions & [node) identifier;
skipWhiteSpaceBeforeNode: runOnce -> { var doNotCallCallbackFunction;}
dontFormatNode: [node) numeric_literal | [node) path ->;
 
-- --------------------------------------------------------------------------------------------------------------------
-- Keep existing whitespace.
-- --------------------------------------------------------------------------------------------------------------------
 
keep_significant_whitespace:
    runOnce
-> {
    var LexerToken = Java.type('oracle.dbtools.parser.LexerToken');
    var tokens = LexerToken.parse(target.input, true);  // include hidden tokens
    var hiddenTokenCount = 0;
    var wsBefore = "";
    var Token = Java.type('oracle.dbtools.parser.Token');
    for (var i in tokens) {
        var type = tokens[i].type;
        if (type == Token.LINE_COMMENT || type == Token.COMMENT || type == Token.WS ||
            type == Token.MACRO_SKIP || type == Token.SQLPLUSLINECONTINUE_SKIP)
        {
            hiddenTokenCount++;
            if (type == Token.WS) {
                wsBefore += tokens[i].content;
            } else {
                wsBefore = "";
            }
        } else {
            if (i-hiddenTokenCount == 0 && hiddenTokenCount == wsBefore.length) {
                struct.putNewline(0, "");
            } else if (wsBefore != " ") {
                struct.putNewline(i-hiddenTokenCount, wsBefore);
            }
            wsBefore = "";
        }
    }
}
 
-- --------------------------------------------------------------------------------------------------------------------
-- Enforce nonquoted identifiers.
-- --------------------------------------------------------------------------------------------------------------------
 
enforce_nonquoted_identifiers:
    runOnce
-> {
    var offOnRanges = [];
 
    var populateOffOnRanges = function(tokens) {
        var off = -1;
        for (var i in tokens) {
            var type = tokens[i].type;
            if (type == Token.LINE_COMMENT || type == Token.COMMENT) {
                if (tokens[i].content.toLowerCase().indexOf("@formatter:off") != -1 
                    || tokens[i].content.toLowerCase().indexOf("noformat start") != -1)
                {
                    off = tokens[i].begin;
                }
                if (off != -1) {
                    if (tokens[i].content.toLowerCase().indexOf("@formatter:on") != -1
                        || tokens[i].content.toLowerCase().indexOf("noformat end") != -1)
                    {
                        offOnRanges.push([off, tokens[i].end]);
                        off = -1;
                    }
                }
            }
        }
    }
 
    var inOffOnRange = function(pos) {
        for (var x in offOnRanges) {
            if (pos >= offOnRanges[x][0] && pos < offOnRanges[x][1]) {
                return true;
            }
        }
        return false;
    }
 
    var HashSet = Java.type('java.util.HashSet');
    var Arrays = Java.type('java.util.Arrays');
    var reservedKeywords = new HashSet(Arrays.asList("ACCESS","ADD","AFTER","ALL","ALLOW","ALTER","ANALYTIC","AND",
        "ANY","ANYSCHEMA","AS","ASC","ASSOCIATE","AUDIT","AUTHID","AUTOMATIC","AUTONOMOUS_TRANSACTION","BEFORE",
        "BEGIN","BETWEEN","BULK","BY","BYTE","CANONICAL","CASE","CASE-SENSITIVE","CHAR","CHECK","CLUSTER","COLUMN",
        "COLUMN_VALUE","COMMENT","COMPOUND","COMPRESS","CONNECT","CONNECT_BY_ROOT","CONSTANT","CONSTRAINT",
        "CONSTRUCTOR","CORRUPT_XID","CORRUPT_XID_ALL","CREATE","CROSSEDITION","CURRENT","CUSTOMDATUM","CYCLE",
        "DATE","DB_ROLE_CHANGE","DECIMAL","DECLARE","DECREMENT","DEFAULT","DEFAULTS","DEFINE","DEFINER","DELETE",
        "DESC","DETERMINISTIC","DIMENSION","DISALLOW","DISASSOCIATE","DISTINCT","DROP","EACH","EDITIONING","ELSE",
        "ELSIF","END","EVALNAME","EXCEPT","EXCEPTION","EXCEPTIONS","EXCEPTION_INIT","EXCLUSIVE","EXISTS","EXTERNAL",
        "FETCH","FILE","FLOAT","FOLLOWING","FOLLOWS","FOR","FORALL","FROM","GOTO","GRANT","GROUP","HAVING","HIDE",
        "HIER_ANCESTOR","HIER_LAG","HIER_LEAD","HIER_PARENT","IDENTIFIED","IF","IGNORE","IMMEDIATE","IMMUTABLE",
        "IN","INCREMENT","INDEX","INDICATOR","INDICES","INITIAL","INITIALLY","INLINE","INSERT","INSTEAD","INTEGER",
        "INTERSECT","INTO","INVISIBLE","IS","ISOLATION","JAVA","JSON_EXISTS","JSON_TABLE","LATERAL","LEVEL","LIBRARY",
        "LIKE","LIKE2","LIKE4","LIKEC","LOCK","LOGON","LONG","MAXEXTENTS","MAXVALUE","MEASURES","MERGE","MINUS",
        "MINVALUE","MLSLABEL","MODE","MODIFY","MULTISET","MUTABLE","NAN","NAV","NCHAR_CS","NESTED_TABLE_ID","NOAUDIT",
        "NOCOMPRESS","NOCOPY","NOCYCLE","NONSCHEMA","NORELY","NOT","NOVALIDATE","NOWAIT","NULL","NUMBER","OF",
        "OFFLINE","ON","ONLINE","ONLY","OPTION","OR","ORADATA","ORDER","ORDINALITY","OVER","OVERRIDING",
        "PARALLEL_ENABLE","PARTITION","PASSING","PAST","PCTFREE","PIPELINED","PIVOT","PRAGMA","PRECEDES",
        "PRECEDING","PRESENT","PRIOR","PROCEDURE","PUBLIC","RAW","REFERENCES","REFERENCING","REJECT","RELY",
        "RENAME","REPEAT","RESOURCE","RESPECT","RESTRICT_REFERENCES","RESULT_CACHE","RETURNING","REVOKE","ROW",
        "ROWID","ROWNUM","ROWS","SELECT","SEQUENTIAL","SERIALIZABLE","SERIALLY_REUSABLE","SERVERERROR","SESSION",
        "SET","SETS","SHARE","SIBLINGS","SINGLE","SIZE","SMALLINT","SOME","SQLDATA","SQL_MACRO","STANDALONE",
        "START","SUBMULTISET","SUBPARTITION","SUCCESSFUL","SUPPRESSES_WARNING_6009","SYNONYM","SYSDATE","TABLE",
        "THE","THEN","TO","TRIGGER","UDF","UID","UNBOUNDED","UNDER","UNION","UNIQUE","UNLIMITED","UNPIVOT","UNTIL",
        "UPDATE","UPSERT","USER","USING","VALIDATE","VALUES","VARCHAR","VARCHAR2","VARRAY","VARYING","VIEW",
        "WHEN","WHENEVER","WHERE","WHILE","WINDOW","WITH","XMLATTRIBUTES","XMLEXISTS","XMLFOREST","XMLNAMESPACES",
        "XMLQUERY","XMLROOT","XMLSCHEMA","XMLSERIALIZE","XMLTABLE"));
 
    var isKeyword = function(token) {
        return reservedKeywords.contains(token.content.replace('"', ""));
    }
 
    var isUnquotingAllowed = function(token) {
        var Pattern = Java.type("java.util.regex.Pattern");
        if (!Pattern.matches('^"[A-Z][A-Z0-9_$#]*"$', token.content)) {
            return false;
        }
        if (isKeyword(token)) {
            return false;
        }
        return true;
    }
 
    var findAndConvertQuotedIdentifiers = function() {
        var tokens = LexerToken.parse(target.input,true);  // include hidden tokens
        populateOffOnRanges(tokens);
        var StringBuilder = Java.type('java.lang.StringBuilder');
        var newInput = new StringBuilder(target.input);
        var delpos = [];
        var conditionalBlock = false;
        for (var i in tokens) {
            var type = tokens[i].type;
            if (type == Token.MACRO_SKIP) {
                var content = tokens[i].content.toLowerCase();
                if (content.indexOf("$if ") == 0) {
                    conditionalBlock = true;
                } else if (content.indexOf("$end") == 0) {
                    conditionalBlock = false;
                }
            }
            if (type == Token.DQUOTED_STRING && isUnquotingAllowed(tokens[i]) 
                && !inOffOnRange(tokens[i].begin) && !conditionalBlock) 
            {
                delpos.push(tokens[i].begin);
                delpos.push(tokens[i].end-1);
            }
        }
        var i = delpos.length - 1;
        while (i >= 0) {
            newInput.deleteCharAt(delpos[i]);
            i--;
        }
        target.input = newInput.toString();
    }
 
    var updateParseTreeAndTokenList = function() {
        var Parsed = Java.type('oracle.dbtools.parser.Parsed');
        var SqlEarley = Java.type('oracle.dbtools.parser.plsql.SqlEarley')
        var defaultTokens = LexerToken.parse(target.input);
        var newTarget = new Parsed(target.input, defaultTokens, SqlEarley.getInstance(), 
            Java.to(["sql_statements"], "java.lang.String[]"));            
        target.src.clear();
        target.src.addAll(newTarget.src);
    }
 
    // main
    findAndConvertQuotedIdentifiers();
    updateParseTreeAndTokenList();
}
 
-- --------------------------------------------------------------------------------------------------------------------
-- Define identifiers (relevant for keyword case and identifier case)
-- --------------------------------------------------------------------------------------------------------------------
 
analytics: [identifier) identifier & [call) analytic_function & [call = [identifier;
ids: [identifier) identifier;
identifiers: ids - analytics ->;

Firstly, the lines 1 to 8 are required by the formatter. They are not interesting in this context.

Secondly, lines 9 to 42 are the heart of a lightweight formatter. This code ensures that all whitespace between all tokens is kept. Therefore, the existing format of the code remains untouched. Read this blog post to learn how SQL Developer’s formatter works.

Thirdly, lines 43 to 173 remove unnecessary double quotes from identifiers. We store the position of double quotes to be removed on lines 147 and 148 in an array named delpos while processing all tokens from start to end. The removal of the double quotes happens on line 153 while processing delpos entries from end to start.

And finally, the lines 174-180 define an Arbori query named identifier. The formatter uses this query to divide lexer tokens of type IDENTIFIER into keywords and identifiers. This is important to ensure that the case of identifiers is left “as is” regardless of the configuration of kwCase.

Doesn’t Connor’s PL/SQL Function Do the Same?

No, when you look closely at the ddl_cleanup.sql script as of 2022-03-02, you will find out that the ddl function has the following limitations:

  • Quoted identifiers are not ignored in
    • Single and multi-line comments
    • Conditional compilation text
    • Code sections for which the formatter is disabled
  • Java Strings are treated as quoted identifiers
  • Reserved keywords are not considered
  • Nonquoted identifiers are changed to lowercase, which might break code using JSON dot notation

It just shows that things become complicated when you don’t solve them in the right place. In this case dbms_metadata’s XSLT scripts. dbms_metadata knows what’s an identifier. It can safely skip the enquoting process if the identifier is in upper case, matches the regular expression ^[A-Z][A-Z0-9_$#]*$ and the identifier is not a reserved keyword. That’s all. The logic can be implemented in a single XSL template. We API users on the other side must parse the code to somehow identify quoted identifiers and their context before we can decide how to proceed.

Formatting DDL Automatically

You can configure SQL Developer to automatically format DDL with your current formatter settings. For that you have to enable the option Autoformat Dictionary Objects SQL as in the screenshot below:

Autoformat DDL

Here’s the result for the deptsal view using the PL/SQL & SQL Formatter Settings:

Autoformat in Action

The identifiers in upper case were originally quoted identifiers. By default, we configure the formatter to keep the case of identifiers. This ensures that code using JSON dot notation is not affected by a formatting operation.

Processing Many Files

SQL Developer is not suited to format many files. However, you can use the SQLcl script or the standalone formatter to format files in a directory tree. The formatter settings (path to the .xml and .arbori file) can be passed as parameters. I recommend using the standalone formatter. It uses the up-to-date and much faster JavaScript engine from GraalVM. Furthermore, the standalone formatter also works with JDK 17, which no longer contains a JavaScript engine.

You can download the latest tvdformat.jar from here. Run java -jar tvdformat.jar to show all command line options.

Summary

If your code base contains generated code, then it probably also contains unnecessarily quoted identifiers. Especially if dbms_metadata was used to extract DDL statements. Removing these double quotes without breaking some code is not that easy. However, SQL Developer’s highly configurable formatter can do the job, even without actually formatting the code.

I hope that some of the shortcomings of dbms_metadata will be addressed in an upcoming release of the Oracle Database. Supporting nonquoted identifiers as an additional non-default option should be easy and not so risky to implement.

Anyway, instead of just detecting violations of G-2180: Never use quoted identifiers, it is a good idea to be able to correct them automatically.

Please open a GitHub issue if you encounter a bug in the formatter settings. Thank you.

The post Quoted Identifiers #JoelKallmanDay appeared first on Philipp Salvisberg's Blog.

]]> Finding Wrong Hints https://www.salvis.com/blog/2022/01/14/finding-wrong-hints/ https://www.salvis.com/blog/2022/01/14/finding-wrong-hints/#comments Fri, 14 Jan 2022 19:32:55 +0000 https://www.salvis.com/blog/?p=11443 Introduction I have used the Oracle Database for many years. And I use hints. For experiments, but also in production code. There are cases when you know more than the Oracle Database. For example about the cardinality of a data source or the number of result rows to process or the number […]

The post Finding Wrong Hints appeared first on Philipp Salvisberg's Blog.

]]> Introduction

I have used the Oracle Database for many years. And I use hints. For experiments, but also in production code. There are cases when you know more than the Oracle Database. For example about the cardinality of a data source or the number of result rows to process or the number of expected executions of a statement. Hints are a way to provide additional information, limit the solution space and enable the database to do a better job. That’s a good thing.

Hints Are Instructions

Hints are passed as special comments at a certain position in SQL statements. They are comments, but they are also instructions. They have to be followed. However, there are cases when hints are not applicable. For example when you request the optimizer to use an index when there is no index defined for the underlying table. In such a case the Oracle Database has basically two options. Either throw an error or ignore the invalid instruction and find another solution. The Oracle Database does the latter.

Hint Report

Starting with version 19c you can produce a hint report that reveals unused hints. Here’s an example:

Unused hint
create table t (c1 integer, c2 varchar2(20));
insert into t values (1, 'one');
insert into t values (2, 'two');
select /*+ index(t) */ * from t where c1 > 0;
select * from dbms_xplan.display_cursor(format => 'basic +hint_report');

EXPLAINED SQL STATEMENT:
------------------------
select /*+ index(t) */ * from t where c1 > 0
 
Plan hash value: 1601196873
 
------------------------------------------
| Id  | Operation                 | Name |
------------------------------------------
|   0 | SELECT STATEMENT          |      |
|   1 |  TABLE ACCESS STORAGE FULL| T    |
------------------------------------------
 
Hint Report (identified by operation id / Query Block Name / Object Alias):
Total hints for statement: 1 (U - Unused (1))
---------------------------------------------------------------------------
 
   1 -  SEL$1 / "T"@"SEL$1"
         U -  index(t)

The hint index(t) defined on line 4 is valid, but it’s reported as unused on line 25. No wonder. There is no index defined on table t.

Let’s create an index and rerun the query.

Used hint
create unique index t_c1_i on t(c1);
select /*+ index(t) */ * from t where c1 > 0;
select * from dbms_xplan.display_cursor(format => 'basic +hint_report');

EXPLAINED SQL STATEMENT:
------------------------
select /*+ index(t) */ * from t where c1 > 0
 
Plan hash value: 2704710798
 
------------------------------------------------------
| Id  | Operation                           | Name   |
------------------------------------------------------
|   0 | SELECT STATEMENT                    |        |
|   1 |  TABLE ACCESS BY INDEX ROWID BATCHED| T      |
|   2 |   INDEX RANGE SCAN                  | T_C1_I |
------------------------------------------------------
 
Hint Report (identified by operation id / Query Block Name / Object Alias):
Total hints for statement: 1
---------------------------------------------------------------------------
 
   1 -  SEL$1 / "T"@"SEL$1"
           -  index(t)

Now the hint index(t) defined on line 2 is reported as used on line 24.

Mixing Hints and Comments

What happens if we mix hints and comments? It depends on where you place the comment. Let’s look at the next example.

Hint followed by unknown tokens
select /*+ index(t) forcing unnecessary index access */ * from t where c1 > 0;
select * from dbms_xplan.display_cursor(format => 'basic +hint_report');

EXPLAINED SQL STATEMENT:
------------------------
select /*+ index(t) forcing unnecessary index access */ * from t where 
c1 > 0
 
Plan hash value: 2704710798
 
------------------------------------------------------
| Id  | Operation                           | Name   |
------------------------------------------------------
|   0 | SELECT STATEMENT                    |        |
|   1 |  TABLE ACCESS BY INDEX ROWID BATCHED| T      |
|   2 |   INDEX RANGE SCAN                  | T_C1_I |
------------------------------------------------------
 
Hint Report (identified by operation id / Query Block Name / Object Alias):
Total hints for statement: 4 (E - Syntax error (3))
---------------------------------------------------------------------------
 
   1 -  SEL$1
         E -  forcing
         E -  index 
         E -  unnecessary
 
   1 -  SEL$1 / "T"@"SEL$1"
           -  index(t)

The comment forcing unnecessary index access on line 1 is interpreted as a series of hints and reported as errors on lines 24 to 26. The token access was not reported. However, the hint index(t) was reported as used on line 29.

What happens if we move the comment to the beginning?

Unknown tokens followed by hint
select /*+ forcing unnecessary index access index(t) */ * from t where c1 > 0;
select * from dbms_xplan.display_cursor(format => 'basic +hint_report');

EXPLAINED SQL STATEMENT:
------------------------
select /*+ forcing unnecessary index access index(t) */ * from t where 
c1 > 0
 
Plan hash value: 2704710798
 
------------------------------------------------------
| Id  | Operation                           | Name   |
------------------------------------------------------
|   0 | SELECT STATEMENT                    |        |
|   1 |  TABLE ACCESS BY INDEX ROWID BATCHED| T      |
|   2 |   INDEX RANGE SCAN                  | T_C1_I |
------------------------------------------------------
 
Hint Report (identified by operation id / Query Block Name / Object Alias):
Total hints for statement: 3 (E - Syntax error (3))
---------------------------------------------------------------------------
 
   1 -  SEL$1
         E -  forcing
         E -  index 
         E -  unnecessary

The same invalid hints are reported as before on lines 24 to 26. However, the hint index(t) was used but not reported as such. This seems to be a limitation of the current hint report in the Oracle Database 21c.

Anyways, it clearly shows that you should not mix comments and hints. Instead, you should write it like this:

Distinguish between comments and hints
select /* forcing unnecessary index access */ /*+ index(t) */ * from t where c1 > 0;
select * from dbms_xplan.display_cursor(format => 'basic +hint_report');

EXPLAINED SQL STATEMENT:
------------------------
select /* forcing unnecessary index access */ /*+ index(t) */ * from t 
where c1 > 0
 
Plan hash value: 2704710798
 
------------------------------------------------------
| Id  | Operation                           | Name   |
------------------------------------------------------
|   0 | SELECT STATEMENT                    |        |
|   1 |  TABLE ACCESS BY INDEX ROWID BATCHED| T      |
|   2 |   INDEX RANGE SCAN                  | T_C1_I |
------------------------------------------------------
 
Hint Report (identified by operation id / Query Block Name / Object Alias):
Total hints for statement: 1
---------------------------------------------------------------------------
 
   1 -  SEL$1 / "T"@"SEL$1"
           -  index(t)

Now the hint index(t) is reported as used. All good, right?

The Problem

I like statically-type languages. Mainly because errors are reported at compile time whenever possible. However, to check hints I need to produce an explain plan. This is possible for a single statement only. This is cumbersome especially when you write code in PL/SQL. As far as I know, there is no option to produce a compile error for invalid hints.

I recently reviewed a system and found a lot of invalid hints. Here are some real-life hints copied from a production code base:

  • /*+ parallel 4 */
  • /*+ no_xml_query_rewrite +materialize */
  • /*+ materialized */
  • /*+ first rows cardinality (a,10) */
  • /*+ append nologging */
  • /*+ le ading(g) u se_nl(g) u se_hash(p, b) */

The last example is a kind of commented-out hint series. In this case, it’s clearly commented-out code. But if you see just a single hint like /*+ le ading(g) */ in the code, you do not know if the space after le was entered intentionally or by accident.

So, how can we identify invalid hints in our code?

Step 1 – Distinguish Between Comments and Hints

We can configure Oracle’s SQL Developer to show hints in a different colour than comments. Here’s the screenshot of an example I showed above:

Distinguish between comments and hints

Go to this GitHub repository and follow the instructions to configure your SQL Developer installation accordingly. See also this blog post for more information about the Arbori code that makes such code highlighting possible.

This step make hints stand out in your code. However, it does not reveal invalid hints.

Step 2 – Install db* CODECOP for SQL Developer

To reveal invalid hints we need a linter. A tool that does some static code analysis. db* CODECOP is such a tool suite. The SQL Developer extension is available for free. It checks the editor content for violations of the Trivadis PL/SQL & SQL Coding Guidelines. Furthermore, db* CODECOP allows you to implement custom guideline checks. The example GitHub repository provides the following four guideline checks regarding hints:

  • G-9600: Never define more than one comment with hints.
  • G-9601: Never use unknown hints.
  • G-9602: Always use the alias name instead of the table name.
  • G-9603: Never reference an unknown table/alias.

To install db* CODECOP and these additional custom guideline checks follow the instructions in this GitHub repository.

Finding Wrong Hints With db* CODECOP

I asked my followers on Twitter if this hint is valid:

Twitter Poll

The result is not really representative. However, 25% thought that /*+ +materialize */ is a valid hint.

Checking the code with db* CODECOP reveals that the hint is invalid and the majority of the poll participants were right.

Invalid hint

Verify Result

But is the result of db* CODECOP correct? The following explain plan shows that the hint /*+ +materialize */ is not reported at all. It’s treated as a comment. Another example is where the hint report is incomplete.

Hint report for +materialize
with e as (
   select /*+ +materialize */ *
     from emp
    where deptno = 10
)
select *
  from e;
select * from dbms_xplan.display_cursor(format => 'basic +hint_report');

EXPLAINED SQL STATEMENT:
------------------------
with e as (    select /*+ +materialize */ *      from emp     where 
deptno = 10 ) select *   from e
 
Plan hash value: 3956160932
 
------------------------------------------
| Id  | Operation                 | Name |
------------------------------------------
|   0 | SELECT STATEMENT          |      |
|   1 |  TABLE ACCESS STORAGE FULL| EMP  |
------------------------------------------

Let’s run the same query after removing the extra + in the hint:

Hint report for materialize
with e as (
   select /*+ materialize */ *
     from emp
    where deptno = 10
)
select *
  from e;
select * from dbms_xplan.display_cursor(format => 'basic +hint_report');

EXPLAINED SQL STATEMENT:
------------------------
with e as (    select /*+ materialize */ *      from emp     where 
deptno = 10 ) select *   from e
 
Plan hash value: 3494145522
 
--------------------------------------------------------------------------------
| Id  | Operation                                | Name                        |
--------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                         |                             |
|   1 |  TEMP TABLE TRANSFORMATION               |                             |
|   2 |   LOAD AS SELECT (CURSOR DURATION MEMORY)| SYS_TEMP_DFD9DB186_8AAEBD74 |
|   3 |    TABLE ACCESS STORAGE FULL             | EMP                         |
|   4 |   VIEW                                   |                             |
|   5 |    TABLE ACCESS STORAGE FULL             | SYS_TEMP_DFD9DB186_8AAEBD74 |
--------------------------------------------------------------------------------
 
Hint Report (identified by operation id / Query Block Name / Object Alias):
Total hints for statement: 1
---------------------------------------------------------------------------
 
   2 -  SEL$1
           -  materialize

Now, the materialize hint has an effect on the execution plan and the hint is reported as used on line 33.

Conclusion

I believe that hints are required for certain use cases. You may have a different opinion. However, if you are using hints in your code you should ensure that they are valid. db* CODECOP can help you to do that. The SQL Developer extension is free. Just use it.

The post Finding Wrong Hints appeared first on Philipp Salvisberg's Blog.

]]> https://www.salvis.com/blog/2022/01/14/finding-wrong-hints/feed/ 8 GraalVM Native Image – First Impressions https://www.salvis.com/blog/2021/12/29/graalvm-native-image-first-impressions/ Wed, 29 Dec 2021 04:43:02 +0000 https://www.salvis.com/blog/?p=11336 Introduction A native image is an operating system-specific executable file. You can build such an image for basically every application running on a Java virtual machine. This approach promises faster start-up times and lower resource consumption. This makes it appealing for serverless computing, auto-scaling platforms and command-line tools. I gained some impressions […]

The post GraalVM Native Image – First Impressions appeared first on Philipp Salvisberg's Blog.

]]> Introduction

A native image is an operating system-specific executable file. You can build such an image for basically every application running on a Java virtual machine. This approach promises faster start-up times and lower resource consumption. This makes it appealing for serverless computing, auto-scaling platforms and command-line tools.

I gained some impressions of this GraalVM technology while developing a standalone command line tool for formatting PL/SQL and SQL code. In this blog post, I share some personal experiences and thoughts.

Starting Point

My starting point is an executable JAR.  I can run it from the command line via java -jar tvdformat.jar. The main class com.trivadis.plsql.formatter.TvdFormat calls a JavaScript format.js and passes all command line parameters to the JavaScript. Behind the scenes, Oracle’s parser and formatter which are part of SQLcl and SQL Developer do the heavy lifting.

It’s quite obvious that this Java application loads a lot of classes and resources dynamically. The GraalVM’s native image builder can identify such objects with the tracing agent. Using the agent is simple. You start the Java application with an additional parameter. The idea is to run the application long enough to detect all dynamically loaded classes and resources. Technically, the trace agent intercepts the calls involved in that dynamic loading process. It’s a best-effort approach. It cannot guarantee completeness.

The next command shows how I run the formatter with the tracing agent for a small PL/SQL project:

Calling tvdformat with tracing agent
java -agentlib:native-image-agent=config-output-dir=config \
     -jar tvdformat.jar $HOME/github/plscope-utils \
     xml=$HOME/github/trivadis/plsql-formatter-settings/settings/sql_developer/trivadis_advanced_format.xml \
     arbori=$HOME/github/trivadis/plsql-formatter-settings/settings/sql_developer/trivadis_custom_format.arbori

This command formats 56 files and the trace agent produces 6 JSON configuration files in the config directory.


jni-config.json

Configuration file for parameter -H:JNIConfigurationFiles. See documentation.

jni-config.json
[
{
  "name":"com.trivadis.plsql.formatter.TvdFormat",
  "methods":[{"name":"main","parameterTypes":["java.lang.String[]"] }]}
,
{
  "name":"java.lang.ClassLoader",
  "methods":[
    {"name":"getPlatformClassLoader","parameterTypes":[] }, 
    {"name":"loadClass","parameterTypes":["java.lang.String"] }
  ]}
,
{
  "name":"java.lang.String",
  "methods":[
    {"name":"lastIndexOf","parameterTypes":["int"] }, 
    {"name":"substring","parameterTypes":["int"] }
  ]}
,
{
  "name":"java.lang.System",
  "methods":[
    {"name":"getProperty","parameterTypes":["java.lang.String"] }, 
    {"name":"setProperty","parameterTypes":["java.lang.String","java.lang.String"] }
  ]}
,
{
  "name":"jdk.internal.loader.ClassLoaders$PlatformClassLoader"}
,
{
  "name":"org.graalvm.nativebridge.jni.JNIExceptionWrapperEntryPoints",
  "methods":[{"name":"getClassName","parameterTypes":["java.lang.Class"] }]}
 
]


predefined-classes-config.json

Configuration file for parameter -H:PredefinedClassesConfigurationFiles.

predefined-classes-config.json
[
  {
    "type":"agent-extracted",
    "classes":[
    ]
  }
]


proxy-config.json

Configuration file for parameter -H:DynamicProxyConfigurationFiles. See documentation.

proxy-config.json
[
]


reflect-config.json

Configuration file for parameter --H:ReflectionConfigurationFiles. See documentation.

reflect-config.json
[
{
  "name":"oracle.dbtools.app.Format",
  "methods":[
    {"name":"breaksAfterComma","parameterTypes":[] }, 
    {"name":"breaksAfterConcat","parameterTypes":[] }, 
    {"name":"breaksAfterLogicalConjunction","parameterTypes":[] }, 
    {"name":"breaksBeforeComma","parameterTypes":[] }, 
    {"name":"breaksBeforeConcat","parameterTypes":[] }, 
    {"name":"breaksBeforeLogicalConjunction","parameterTypes":[] }, 
    {"name":"dontFormatNode","parameterTypes":["oracle.dbtools.parser.Parsed","java.util.Map"] }, 
    {"name":"identifiers","parameterTypes":["oracle.dbtools.parser.Parsed","java.util.Map"] }, 
    {"name":"indentConditions","parameterTypes":[] }
  ]}
,
{
  "name":"oracle.dbtools.app.Persist2XML",
  "methods":[{"name":"values","parameterTypes":["oracle.dbtools.parser.Parsed","java.util.Map"] }]}
 
]


resource-config.json

Configuration file for parameter -H:ResourceConfigurationFiles. See documentation.

resource-config.json
{
  "resources":{
  "includes":[
    {
      "pattern":"\\QMETA-INF/services/com.oracle.truffle.api.TruffleLanguage$Provider\\E"
    }, 
    {
      "pattern":"\\QMETA-INF/services/com.oracle.truffle.api.instrumentation.TruffleInstrument$Provider\\E"
    }, 
    {
      "pattern":"\\QMETA-INF/services/com.oracle.truffle.api.library.DefaultExportProvider\\E"
    }, 
    {
      "pattern":"\\QMETA-INF/services/com.oracle.truffle.api.object.LayoutFactory\\E"
    }, 
    {
      "pattern":"\\QMETA-INF/services/com.oracle.truffle.js.runtime.Evaluator\\E"
    }, 
    {
      "pattern":"\\QMETA-INF/services/java.nio.file.spi.FileSystemProvider\\E"
    }, 
    {
      "pattern":"\\QMETA-INF/services/javax.script.ScriptEngineFactory\\E"
    }, 
    {
      "pattern":"\\QMETA-INF/services/org.graalvm.polyglot.impl.AbstractPolyglotImpl\\E"
    }, 
    {
      "pattern":"\\Qcom/oracle/truffle/nfi/backend/libffi/LibFFILanguage.class\\E"
    }, 
    {
      "pattern":"\\Qformat.js\\E"
    }, 
    {
      "pattern":"\\Qoracle/dbtools/app/format.prg\\E"
    }, 
    {
      "pattern":"\\Qoracle/dbtools/app/persist2xml.prg\\E"
    }, 
    {
      "pattern":"\\Qoracle/dbtools/arbori/arbori.grammar\\E"
    }, 
    {
      "pattern":"\\Qoracle/dbtools/arbori/std.arbori\\E"
    }, 
    {
      "pattern":"\\Qoracle/dbtools/parser/js/js.grammar\\E"
    }, 
    {
      "pattern":"\\Qoracle/dbtools/parser/plsql/allRules.txt\\E"
    }, 
    {
      "pattern":"\\Qoracle/dbtools/parser/plsql/doc/frequencies.txt\\E"
    }, 
    {
      "pattern":"\\Qoracle/dbtools/parser/xml.grammar\\E"
    }
  ]},
  "bundles":[{
      "name":"oracle.dbtools.util.Messages"
    }]
}


serialization-config.json

Configuration file for parameter -H:SerializationConfigurationFiles. “The serialization support ensures constructors for classes are contained in a native image so that they can be deserialized in the first place”. See release notes of GraalVM 21.0.0.

serialization-config.json
[
]


Environment

The environment for this experiment was:

  • MacBook Pro (16-inch, 2021) with an Apple M1 Max chip and 64 GB memory running on macOS Monterey 12.0.1
  • GraalVM CE 21.3.0 (build 17.0.1+12-jvmci-21.3-b05)
  • Apache Maven 3.8.3
  • SQLcl: Release 21.4.0.0 Production Build: 21.4.0.348.1716, installed in /usr/local/bin/sqlcl
  • Standalone PL/SQL & SQL Formatter at commit b4d26bd installed in $HOME/trivadis/plsql-formatter-settings
    • tvdformat.jar produced via mvn -DskipTests=true package in the standalone/target subdirectory
    • zip -d tvdformat-21.4.1-SNAPSHOT.jar "META-INF/native-image/*" to remove the native-image configuration files (they would be automatically used otherwise)
  • plscope-utils at commit 0687f5c installed in $HOME/github/plscope-utils

You find the configuration files used in this blog post in this Gist.

Building Image With Tracing Agent’s Config Files

Let’s try to build a native image with these configuration files.

Build native image (1)
$JAVA_HOME/bin/native-image \
-cp /usr/local/bin/sqlcl/lib/dbtools-common.jar:\
tvdformat-21.4.1-SNAPSHOT.jar \
-H:JNIConfigurationFiles=config/jni-config.json \
-H:PredefinedClassesConfigurationFiles=config/predefined-classes-config.json \
-H:DynamicProxyConfigurationFiles=config/proxy-config.json \
-H:ReflectionConfigurationFiles=config/reflect-config.json \
-H:ResourceConfigurationFiles=config/resource-config.json \
-H:SerializationConfigurationFiles=config/serialization-config.json \
-H:+ReportExceptionStackTraces \
--language:js \
-H:Class=com.trivadis.plsql.formatter.TvdFormat \
-H:Name=tvdformat

Here’s the console output:

Console output
[tvdformat:15574]    classlist:   3,420.59 ms,  0.96 GB
[tvdformat:15574]        (cap):   2,147.77 ms,  0.96 GB
[tvdformat:15574]        setup:   6,412.44 ms,  0.96 GB
[tvdformat:15574]     (clinit):   1,036.63 ms,  6.18 GB
[tvdformat:15574]   (typeflow):  16,832.01 ms,  6.18 GB
[tvdformat:15574]    (objects):  31,681.96 ms,  6.18 GB
[tvdformat:15574]   (features):  12,333.23 ms,  6.18 GB
[tvdformat:15574]     analysis:  63,801.37 ms,  6.18 GB
[tvdformat:15574]     universe:   3,216.66 ms,  6.18 GB
10971 method(s) included for runtime compilation
[tvdformat:15574]      (parse):  11,562.89 ms,  6.14 GB
[tvdformat:15574]     (inline):   6,112.92 ms,  7.19 GB
[tvdformat:15574]    (compile):  35,886.56 ms,  7.23 GB
[tvdformat:15574]      compile:  58,027.14 ms,  7.07 GB
[tvdformat:15574]        image:   5,407.78 ms,  7.07 GB
[tvdformat:15574]        write:   2,973.57 ms,  7.07 GB
[tvdformat:15574]      [total]: 147,632.30 ms,  7.07 GB
# Printing build artifacts to: /Users/phs/github/trivadis/plsql-formatter-settings/standalone/target/tvdformat.build_artifacts.txt

No error messages. Great. And what’s the size of the tvdformat executable? 115 MB. The --language:js parameter contributes about 98 MB, which includes probably a bit more than necessary.

Anyway, let’s run the native image.

Run tvdformat as native image
./tvdformat $HOME/github/plscope-utils \
xml=$HOME/github/trivadis/plsql-formatter-settings/settings/sql_developer/trivadis_advanced_format.xml \
arbori=$HOME/github/trivadis/plsql-formatter-settings/settings/sql_developer/trivadis_custom_format.arbori

This call produces the following console output:

Plaintext
Exception in thread "main" javax.script.ScriptException: org.graalvm.polyglot.PolyglotException: TypeError: Access to host class java.lang.String is not allowed or does not exist.
	at com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.toScriptException(GraalJSScriptEngine.java:483)
	at com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:460)
	at com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:400)
	at com.trivadis.plsql.formatter.TvdFormat.run(TvdFormat.java:34)
	at com.trivadis.plsql.formatter.TvdFormat.main(TvdFormat.java:67)
Caused by: org.graalvm.polyglot.PolyglotException: TypeError: Access to host class java.lang.String is not allowed or does not exist.
	at <js>.:program(<eval>:23)
	at org.graalvm.polyglot.Context.eval(Context.java:379)
	at com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:458)
	... 3 more

The String class is used on line 23 in format.js. We need to register Java classes used in JavaScript and extend the configuration accordingly.

Extending Reflection Configuration (1)

I reviewed the format.js  and the JavaScript callback functions in trivadis_custom_format.arbori and created an additional configuration file reflect-config2.json for all Java classes used in JavaScript.


reflect-config2.json

[
  {
    "name" : "com.trivadis.plsql.formatter.ScriptRunnerContext",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.io.BufferedWriter",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.io.File",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.io.PrintStream",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.lang.Class",
    "methods" : [
      {"name" : "getDeclaredField"}
    ]
  },
  {
    "name" : "java.lang.Integer",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.lang.reflect.Field",
    "methods" : [
      { "name" : "setAccessible"},
      { "name" : "get"}
    ]
  },
  {
    "name" : "java.lang.String",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.lang.StringBuilder",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.lang.System",
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.net.URI",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.nio.file.Files",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.nio.file.FileSystems",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.nio.file.Paths",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.util.ArrayList",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.util.Arrays",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.util.Collection",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.util.HashMap",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.util.HashMap$KeySet",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.util.HashSet",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.util.Iterator",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.util.regex.Matcher",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.util.regex.Pattern",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.util.stream.Collectors",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.util.stream.ReferencePipeline$Head",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.util.stream.Stream",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "java.util.TreeSet",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.app.Format",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.app.Format$Breaks",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.app.Format$BreaksX2",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.app.Format$Case",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.app.Format$FlowControl",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.app.Format$InlineComments",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.app.Format$Space",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.app.Persist2XML",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.arbori.Program",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.arbori.SqlProgram",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.parser.Lexer",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.parser.LexerToken",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.parser.Parsed",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.parser.ParseNode",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.parser.plsql.SqlEarley",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.parser.plsql.SyntaxError",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.parser.Substitutions",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.parser.Token",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.util.Logger",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  },
  {
    "name" : "oracle.dbtools.util.Service",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allDeclaredClasses": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true,
    "allPublicClasses": true
  }
]


Now we can build the native image with this additional configuration file.

Build native image (2)
$JAVA_HOME/bin/native-image \
-cp /usr/local/bin/sqlcl/lib/dbtools-common.jar:\
tvdformat-21.4.1-SNAPSHOT.jar \
-H:JNIConfigurationFiles=config/jni-config.json \
-H:PredefinedClassesConfigurationFiles=config/predefined-classes-config.json \
-H:DynamicProxyConfigurationFiles=config/proxy-config.json \
-H:ReflectionConfigurationFiles=config/reflect-config.json,config/reflect-config2.json \
-H:ResourceConfigurationFiles=config/resource-config.json \
-H:SerializationConfigurationFiles=config/serialization-config.json \
-H:+ReportExceptionStackTraces \
--language:js \
-H:Class=com.trivadis.plsql.formatter.TvdFormat \
-H:Name=tvdformat

The build completes without errors and produces a native image of 138 MB. 23 MB larger. Let’s run it.

The second run produces this console output:

Console output of tvdformat run (2)
Exception in thread "main" javax.script.ScriptException: org.graalvm.polyglot.PolyglotException: com.oracle.svm.core.jdk.UnsupportedFeatureError: Proxy class defined by interfaces [interface java.util.function.Predicate] not found. Generating proxy classes at runtime is not supported. Proxy classes need to be defined at image build time by specifying the list of interfaces that they implement. To define proxy classes use -H:DynamicProxyConfigurationFiles=<comma-separated-config-files> and -H:DynamicProxyConfigurationResources=<comma-separated-config-resources> options.
	at com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.toScriptException(GraalJSScriptEngine.java:483)
	at com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:460)
	at com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:400)
	at com.trivadis.plsql.formatter.TvdFormat.run(TvdFormat.java:34)
	at com.trivadis.plsql.formatter.TvdFormat.main(TvdFormat.java:67)

An excellent error message. We need to configure the class java.util.function.Predicate via -H:DynamicProxyConfigurationFiles. Let’s do this.

Extending Dynamic Proxy Configuration

For this blog post, I decided to create a second configuration file proxy-config2.json to distinguish it from the one generated by the trace agent.

proxy-config2.json
[
  ["java.util.function.Predicate"]
]

Let’s build the native image with this additional configuration file.

Build native image (3)
$JAVA_HOME/bin/native-image \
-cp /usr/local/bin/sqlcl/lib/dbtools-common.jar:\
tvdformat-21.4.1-SNAPSHOT.jar \
-H:JNIConfigurationFiles=config/jni-config.json \
-H:PredefinedClassesConfigurationFiles=config/predefined-classes-config.json \
-H:DynamicProxyConfigurationFiles=config/proxy-config.json,config/proxy-config2.json \
-H:ReflectionConfigurationFiles=config/reflect-config.json,config/reflect-config2.json \
-H:ResourceConfigurationFiles=config/resource-config.json \
-H:SerializationConfigurationFiles=config/serialization-config.json \
-H:+ReportExceptionStackTraces \
--language:js \
-H:Class=com.trivadis.plsql.formatter.TvdFormat \
-H:Name=tvdformat

The build completes without errors and produces a native image of 138 MB. The same size as before. Let’s run it.

The third run produces this console output:

Console output of tvdformat run (3)
Formatting file 1 of 56: /Users/phs/github/plscope-utils/README.md... done.
Formatting file 2 of 56: /Users/phs/github/plscope-utils/database/README.md... Exception in thread "main" javax.script.ScriptException: java.lang.Exception: java.lang.AssertionError: oracle.dbtools.arbori.ScriptException: java.lang.NumberFormatException: Cannot parse null string
	at oracle.dbtools.app.Format.format(Format.java:387)
	at <js>.formatMarkdownFile(<eval>:489)
	at <js>.formatFiles(<eval>:528)
	at <js>.run(<eval>:552)
	at <js>.:program(<eval>:617)
	at org.graalvm.polyglot.Context.eval(Context.java:379)
	at com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:458)
	at com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:400)
	at com.trivadis.plsql.formatter.TvdFormat.run(TvdFormat.java:34)
	at com.trivadis.plsql.formatter.TvdFormat.main(TvdFormat.java:67)
Caused by: java.lang.Exception: java.lang.AssertionError: oracle.dbtools.arbori.ScriptException: java.lang.NumberFormatException: Cannot parse null string
	at com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.toScriptException(GraalJSScriptEngine.java:476)
	at com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:460)
	... 3 more

This is quite interesting. The embedded format.js works now. The first file README.md does not contain SQL text blocks. and therefore the formatter was not called and no error was reported. However, the formatter failed for the second file. What could be the reason for NumberFormatException: Cannot parse null string? – In this case the ParseNode class could not be loaded dynamically. To load this class successfully a lot of other classes are also required.

Extending Reflection Configuration (2)

Identifying all the dynamically loaded classes is not that simple. To debug the native image you can enable or add logging output, review the related source code or use a debugger. The native image debugger is an enterprise feature that is on the roadmap for the community edition. However, you still need to identify the reason for every single runtime exception. After adding the class to the configuration file you need to rebuild the native image and run it to detect the next exception. Doing this manually is really time-consuming.

Another approach is to register classes and their constructors, methods and fields programmatically using configuration with features. I’ve done that for a few chosen packages of the dbtools-common.jar that are part of the SQLcl installation. See source on GitHub.

For this blog post, I created an additional configuration file with 218 classes.


reflect-config3.json

[
  {
    "name": "oracle.dbtools.app.CompletionItem",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.CompletionItem$Type",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.CompletionList",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.Format",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.Format$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.Format$2",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.Format$3",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.Format$Breaks",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.Format$BreaksX2",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.Format$Case",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.Format$FlowControl",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.Format$InlineComments",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.Format$Space",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.Obfuscator",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.Persist2XML",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.Rewrite",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.SqlCompleter$IProgram",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.SqlId",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.XML2Table",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.XML2Table$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.CallLink$State",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.Dotted",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.FlowGraph$Debug",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.Loc",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.Loc$LocType",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.PlsqlException",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.PlsqlException$Cause",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.PlsqlType",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.PlsqlType$CharFamily",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.PlsqlType$Collection",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.PlsqlType$DateFamily",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.PlsqlType$Ellipsis",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.PlsqlType$Exception",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.PlsqlType$Function",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.PlsqlType$Function$ARGMATCH",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.PlsqlType$NumericFamily",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.PlsqlType$Record",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.PlsqlType$Scalar",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.PlsqlType$ScalarClass",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.PlsqlType$ScalarType",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.PlsqlType$Scope",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.PlsqlType$TYPECLASS",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.RegressionTest$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SqlInjection",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SqlInjection$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SqlInjection$2",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SqlInjectionAnalysisFailure",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SqlInjectionAnalysisFailure$SqlInjectionFailureTypes",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SqlInjectionGraph$Debug",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SqlInjectionGraph$ParameterMode",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SqlInjectionGraph$PickTokens",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SqlInjectionGraph$Reduction",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SqlInjectionGraph$Returns",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SqlInjectionGraph$SubprgBodyState",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SqlInjectionGraph$Token",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SqlInjectionGraph$Tokens",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SqlInjectionWarningTest$Issue",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SqlInjectionWarningTest$Issue$Type",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SqlInjectionWarningTest$SqlInjectionAdvice",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.Symbol",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.Symbol$FunctionSym",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.Symbol$ObjectSym",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.Symbol$VariableSym",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SymbolTable",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SymbolTable$RecordSym",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.SymbolTable$ScopeType",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.ValueNode",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.ValueNode$ArgNode",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.ValueNode$Debug",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.ValueNode$DottedNode",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.ValueNode$ExprNode",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.ValueNode$FunctionNodeDeclaration",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.ValueNode$IdentifierNode",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.ValueNode$IdentifierNodeDeclaration",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.ValueNode$LHSNode",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.ValueNode$LiteralNode",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.ValueNode$ModeNode",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.ValueNode$MultiArgNode",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.ValueNode$MultiExprNode",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.ValueNode$MultiIdentifierNodeDeclaration",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.ValueNode$MultiNode",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.ValueNode$TokenNode",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.ValueNode$TypeNode",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.ValueNode$ValueType",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.app.injection.VerifyValueNodeTypes$Debug",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.AggregatePredicate",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.AggregatePredicate$Type",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.AncestorDescendantNodes",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.AncestorDescendantNodes$Type",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.AncestorExpr",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Attribute",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.AttributeDefinitions",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.BindVar",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.ChildExpr",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.ChildNumRelation",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Column",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Composite",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.CompositeExpr",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.EqualExpr",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.False",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Head",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Header",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.IdentedPredicate",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.IndependentAttribute",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.JSFunc",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.MaterializedPredicate",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.NodeContent",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.NodeMatchingSrc",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.NodesWMatchingSrc",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Oper",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Parent",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.ParentChildNodes",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Position",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.PositionalRelation",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.PredRef",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Predicate",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Program",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Program$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Program$2",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Program$Command",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Program$IncludedPrg",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Program$JavaScriptBlock",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Program$PredicateCmd",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.SameNodes",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.ScriptException",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Sibling",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.SqlProgram",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Tail",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.True",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.arbori.Tuple",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Cell",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.EBNF$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Earley",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Earley$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Earley$PredictedTerminals",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Grammar$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Grammar$2",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.LeafNode",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.LexerRegressionTest$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Matriceable",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Matrix",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.ParseNode",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Parseable",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Parsed",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Parser",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Parser$EarleyCell",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Parser$EarleyCell$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Parser$Tuple",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.RecognizedRule",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.RuleTuple",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Token",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Visual",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Visual$1ScrollablePicture",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Visual$1ScrollablePicture$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Yelrae",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.Yelrae$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.dictionary.DictionaryEarley$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.html.HtmlEarley$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.html.HtmlParser",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.html.HtmlParser$Identifier",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.html.HtmlParser$Keyword",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.html.HtmlParser$Optional",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.html.HtmlParser$ParsedGrammarSymbol",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.html.HtmlParser$Sequence",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.js.JavaScript$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.json.Interpreter$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.json.JsonEarley$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.json.ResponseError",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.plsql.BasicSuggestedItem",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.plsql.BasicTabCol$Column",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.plsql.LazyNode",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.plsql.ParsedSql",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.plsql.ParsedSql$Language",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.plsql.SqlEarley",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.plsql.SqlEarley$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.plsql.SqlRules$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.plsql.StackParser",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.plsql.StackParser$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.plsql.SyntaxError",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.plsql.SyntaxError$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.plsql.SyntaxError$2",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.plsql.SyntaxError$3",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.plsql.doc.DocURL",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.plsql.doc.DocURL$Manual",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.plsql.doc.HarvestDoc$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.plsql.doc.Substr",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.parser.translation.Translate$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.raptor.LazyIcon",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.Encodings$CharsetComparator",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.InputOutputStreams$EmptyStream",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.InputOutputStreams$NullOutputStream",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.InputOutputStreams$Uncloseable",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.KeyboardFocusLogger",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.LogHandler",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.LogHandler$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.Messages",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.Service$1",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.Service$2",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.Service$3",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.Service$CLEAN_TYPE",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.StreamCopy$EmptyStream",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.VacuousCallableStatement",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.VacuousParameterMetaData",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.VacuousPreparedStatement",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.VacuousStatement",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.VacuousStatement$VacuousResultSet",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.VacuousStatement$VacuousResultSet$VacuousResultSetMetaData",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.encoding.BASE64Encoding",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.encoding.Decoder",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.encoding.Encoder",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.encoding.Encoding",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.encoding.EncodingException",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.encoding.EncodingType",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.encoding.HEXEncoding",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.encoding.MimeType",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.encoding.NullEncoding",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.encoding.StorageType",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  },
  {
    "name": "oracle.dbtools.util.encoding.URIEncoding",
    "allDeclaredConstructors": true,
    "allDeclaredMethods": true,
    "allDeclaredFields": true,
    "allPublicConstructors": true,
    "allPublicMethods": true,
    "allPublicFields": true
  }
]


Let’s once more build the native image with an additional configuration file.

Build native image (4)
$JAVA_HOME/bin/native-image \
-cp /usr/local/bin/sqlcl/lib/dbtools-common.jar:\
tvdformat-21.4.1-SNAPSHOT.jar \
-H:JNIConfigurationFiles=config/jni-config.json \
-H:PredefinedClassesConfigurationFiles=config/predefined-classes-config.json \
-H:DynamicProxyConfigurationFiles=config/proxy-config.json,config/proxy-config2.json \
-H:ReflectionConfigurationFiles=\
config/reflect-config.json,config/reflect-config2.json,config/reflect-config3.json \
-H:ResourceConfigurationFiles=config/resource-config.json \
-H:SerializationConfigurationFiles=config/serialization-config.json \
-H:+ReportExceptionStackTraces \
--language:js \
-H:Class=com.trivadis.plsql.formatter.TvdFormat \
-H:Name=tvdformat

The build completes without errors and produces a native image of 145 MB. 7 MB larger than before. Let’s run it.

The fourth run produces this console output:

Console output of tvdformat run (4)
Formatting file 1 of 56: /Users/phs/github/plscope-utils/README.md... done.
Formatting file 2 of 56: /Users/phs/github/plscope-utils/database/README.md... done.
Formatting file 3 of 56: /Users/phs/github/plscope-utils/database/demo/demo_script/00_demo_readme.sql... done.
Formatting file 4 of 56: /Users/phs/github/plscope-utils/database/demo/demo_script/01_demo_plscope.sql... done.
Formatting file 5 of 56: /Users/phs/github/plscope-utils/database/demo/demo_script/02_demo_lineage.sql... done.
Formatting file 6 of 56: /Users/phs/github/plscope-utils/database/demo/demo_script/03_demo_utl_xml_parsequery.sql... done.
Formatting file 7 of 56: /Users/phs/github/plscope-utils/database/demo/package/etl.pkb... done.
Formatting file 8 of 56: /Users/phs/github/plscope-utils/database/demo/package/etl.pks... done.
Formatting file 9 of 56: /Users/phs/github/plscope-utils/database/demo/synonym/source_syn.sql... done.
Formatting file 10 of 56: /Users/phs/github/plscope-utils/database/demo/table/dept.sql... done.
Formatting file 11 of 56: /Users/phs/github/plscope-utils/database/demo/table/deptsal.sql... done.
Formatting file 12 of 56: /Users/phs/github/plscope-utils/database/demo/table/deptsal_err.sql... done.
Formatting file 13 of 56: /Users/phs/github/plscope-utils/database/demo/table/drop_demo_tables.sql... done.
Formatting file 14 of 56: /Users/phs/github/plscope-utils/database/demo/table/emp.sql... done.
Formatting file 15 of 56: /Users/phs/github/plscope-utils/database/demo/view/source_view.sql... done.
Formatting file 16 of 56: /Users/phs/github/plscope-utils/database/install.sql... done.
Formatting file 17 of 56: /Users/phs/github/plscope-utils/database/install_test.sql... done.
Formatting file 18 of 56: /Users/phs/github/plscope-utils/database/test/package/test_dd_util.pkb... done.
Formatting file 19 of 56: /Users/phs/github/plscope-utils/database/test/package/test_dd_util.pks... done.
Formatting file 20 of 56: /Users/phs/github/plscope-utils/database/test/package/test_etl.pkb... done.
Formatting file 21 of 56: /Users/phs/github/plscope-utils/database/test/package/test_etl.pks... done.
Formatting file 22 of 56: /Users/phs/github/plscope-utils/database/test/package/test_lineage_util.pkb... done.
Formatting file 23 of 56: /Users/phs/github/plscope-utils/database/test/package/test_lineage_util.pks... done.
Formatting file 24 of 56: /Users/phs/github/plscope-utils/database/test/package/test_parse_util.pkb... done.
Formatting file 25 of 56: /Users/phs/github/plscope-utils/database/test/package/test_parse_util.pks... done.
Formatting file 26 of 56: /Users/phs/github/plscope-utils/database/test/package/test_plscope_context.pkb... done.
Formatting file 27 of 56: /Users/phs/github/plscope-utils/database/test/package/test_plscope_context.pks... done.
Formatting file 28 of 56: /Users/phs/github/plscope-utils/database/test/package/test_plscope_identifiers.pkb... done.
Formatting file 29 of 56: /Users/phs/github/plscope-utils/database/test/package/test_plscope_identifiers.pks... done.
Formatting file 30 of 56: /Users/phs/github/plscope-utils/database/test/package/test_type_util.pkb... done.
Formatting file 31 of 56: /Users/phs/github/plscope-utils/database/test/package/test_type_util.pks... done.
Formatting file 32 of 56: /Users/phs/github/plscope-utils/database/utils/context/plscope.ctx... done.
Formatting file 33 of 56: /Users/phs/github/plscope-utils/database/utils/package/dd_util.pkb... done.
Formatting file 34 of 56: /Users/phs/github/plscope-utils/database/utils/package/dd_util.pks... done.
Formatting file 35 of 56: /Users/phs/github/plscope-utils/database/utils/package/lineage_util.pkb... done.
Formatting file 36 of 56: /Users/phs/github/plscope-utils/database/utils/package/lineage_util.pks... done.
Formatting file 37 of 56: /Users/phs/github/plscope-utils/database/utils/package/parse_util.pkb... done.
Formatting file 38 of 56: /Users/phs/github/plscope-utils/database/utils/package/parse_util.pks... done.
Formatting file 39 of 56: /Users/phs/github/plscope-utils/database/utils/package/plscope_context.pkb... done.
Formatting file 40 of 56: /Users/phs/github/plscope-utils/database/utils/package/plscope_context.pks... done.
Formatting file 41 of 56: /Users/phs/github/plscope-utils/database/utils/package/type_util.pkb... done.
Formatting file 42 of 56: /Users/phs/github/plscope-utils/database/utils/package/type_util.pks... done.
Formatting file 43 of 56: /Users/phs/github/plscope-utils/database/utils/type/col_lineage_type.sql... done.
Formatting file 44 of 56: /Users/phs/github/plscope-utils/database/utils/type/col_type.sql... done.
Formatting file 45 of 56: /Users/phs/github/plscope-utils/database/utils/type/obj_type.sql... done.
Formatting file 46 of 56: /Users/phs/github/plscope-utils/database/utils/type/t_col_lineage_type.sql... done.
Formatting file 47 of 56: /Users/phs/github/plscope-utils/database/utils/type/t_col_type.sql... done.
Formatting file 48 of 56: /Users/phs/github/plscope-utils/database/utils/type/t_obj_type.sql... done.
Formatting file 49 of 56: /Users/phs/github/plscope-utils/database/utils/user/plscope.sql... done.
Formatting file 50 of 56: /Users/phs/github/plscope-utils/database/utils/view/plscope_col_usage.sql... done.
Formatting file 51 of 56: /Users/phs/github/plscope-utils/database/utils/view/plscope_identifiers.sql... done.
Formatting file 52 of 56: /Users/phs/github/plscope-utils/database/utils/view/plscope_ins_lineage.sql... done.
Formatting file 53 of 56: /Users/phs/github/plscope-utils/database/utils/view/plscope_naming.sql... done.
Formatting file 54 of 56: /Users/phs/github/plscope-utils/database/utils/view/plscope_statements.sql... done.
Formatting file 55 of 56: /Users/phs/github/plscope-utils/database/utils/view/plscope_tab_usage.sql... done.
Formatting file 56 of 56: /Users/phs/github/plscope-utils/sqldev/README.md... done.

Finally, a working native image.

Testing

I have a total of 604 JUnit tests for the formatter. When all tests succeed I can expect that the executable JAR works as well.

Now we need additional tests to detect runtime errors that happen only when running a native image. This is necessary for missing configurations as shown above, but also for other cases where the native image behaves differently.

In my case, I want to run three existing JUnit tests for the main method in TvdFormat. How do I do that? A good choice is to use the Maven plugin for GraalVM Native Image building (of course there is also a plugin for Gradle). This plugin produces a dedicated native image for tests. When you run this native image all configured tests are executed and the results are shown on the console. A non-zero exit status indicates a failure.

When you run mvn -Dnative.skip=false integration-test a native image named native-tests is created and executed during the build. You can run it also after the build via ./native-tests. It produces the following console output:

Console output of the native-tests
JUnit Platform on Native Image - report
----------------------------------------

com.trivadis.plsql.formatter.standalone.tests.TvdFormatTest > jsonArrayFileTest() SUCCESSFUL

com.trivadis.plsql.formatter.standalone.tests.TvdFormatTest > jsonArrayDirTest() SUCCESSFUL

com.trivadis.plsql.formatter.standalone.tests.TvdFormatTest > jsonObjectFileTest() SUCCESSFUL


Test run finished after 3198 ms
[         2 containers found      ]
[         0 containers skipped    ]
[         2 containers started    ]
[         0 containers aborted    ]
[         2 containers successful ]
[         0 containers failed     ]
[         3 tests found           ]
[         0 tests skipped         ]
[         3 tests started         ]
[         0 tests aborted         ]
[         3 tests successful      ]
[         0 tests failed          ]

You find these JUnit tests here and the Maven configuration here.

Maybe you wonder why I do not run all JUnit tests. Simply put, they do not work. Some of them intentionally, because they are testing the integration with SQLcl which is not applicable in the standalone image (the necessary libraries are not included by purpose). Other test cases require configuration changes or need to be rewritten for use in native images. In other words, there is still work to be done.

Performance

What to Measure

Let’s compare the runtimes of the following execution variants:

  1. Native image with GraalVM CE 21.3.0 (build 17.0.1+12-jvmci-21.3-b05
  2. Executable JAR with GraalVM CE 21.3.0 (build 17.0.1+12-jvmci-21.3-b05)
  3. Executable JAR with JDK 17.0.1 for macOS ARM 64

The GraalVM JDK is not yet available for macOS ARM 64 (see GitHub issue). This means that we must use the Intel x64 variant, which requires Rosetta 2 to translate the Intel x64 instructions for the M1 Max chip. This emulation works quite well and is transparent for the user. However, it needs time. To get an idea of the performance improvement of the native image technology, option (1) and (2) should be compared. Option (3) is still interesting for comparison with option (2). It shows the impact of the Rosetta 2 chip emulation.

I’d like to measure the performance of these two scenarios:

  1. Startup time (start the executable without parameters to show the help)
  2. Formatting 56 files (of plscope-util project, that is what we’ve run previously)

Scenario 1 – Startup Time

In this scenario, we measure the startup time of the formatter. This means we call the formatter without parameters to show the help. The shell script shows what I measured.


Shell script - startup time

bench1.sh
#!/bin/zsh
 
call() {
    echo "$1" "($2)"
    for ((i=0; i<3; i++))
    do
       eval "time $2 > /dev/null"
    done;
}
 
echo
echo Scenario 1 - Startup Time
echo -------------------------
unset JAVA_HOME
call "1) Native Image" "./tvdformat"
export JAVA_HOME=$HOME/Applications/graalvm-ce-java17-21.3.0/Contents/Home
call "2) GraalVM" "java -jar ./tvdformat.jar"
export JAVA_HOME=`/usr/libexec/java_home -v 17`
call "3) ARM JDK" "java -jar ./tvdformat.jar"
Console output of shell script
Scenario 1 - Startup Time
-------------------------
1) Native Image (./tvdformat)
  0.29s user 0.05s system 94% cpu 0.355 total
  0.29s user 0.04s system 96% cpu 0.336 total
  0.29s user 0.04s system 96% cpu 0.337 total
2) GraalVM (java -jar ./tvdformat.jar)
  2.99s user 0.43s system 143% cpu 2.387 total
  2.97s user 0.42s system 142% cpu 2.376 total
  2.99s user 0.43s system 142% cpu 2.392 total
3) ARM JDK (java -jar ./tvdformat.jar)
  0.65s user 0.04s system 167% cpu 0.414 total
  0.65s user 0.05s system 165% cpu 0.419 total
  0.64s user 0.04s system 159% cpu 0.427 total


I used the result of the second execution per variant to create the chart.

Scenario 1 - Startup Time - Native Image vs Executable JAR

The native image delivers by far the fastest startup times. It’s 7 times faster and uses 10 times less CPU resources. Rosetta 2 leads to an overhead of about factor 4.7 from a CPU usage perspective.

Scenario 2 – Formatting 56 files

In this scenario, we measure the time to format 56 PL/SQL and SQL files of the plscope-util project. The shell script shows what I measured.


Shell script - formatting 56 files

bench2.sh
#!/bin/zsh
 
call() {
    echo "$1" "($2)"
    for ((i=0; i<3; i++))
    do
       eval "time $2 $HOME/github/plscope-utils \
            xml=$HOME/github/trivadis/plsql-formatter-settings/settings/sql_developer/trivadis_advanced_format.xml \
            arbori=$HOME/github/trivadis/plsql-formatter-settings/settings/sql_developer/trivadis_custom_format.arbori \
            > /dev/null"
    done;
}
 
echo
echo Scenario 2 - Formatting 56 files
echo --------------------------------
unset JAVA_HOME
call "1) Native Image" "./tvdformat"
export JAVA_HOME=$HOME/Applications/graalvm-ce-java17-21.3.0/Contents/Home
call "2) GraalVM" "java -jar ./tvdformat.jar"
export JAVA_HOME=`/usr/libexec/java_home -v 17`
call "3) ARM JDK" "java -jar ./tvdformat.jar"
Console output of shell script
Scenario 2 - Formatting 56 files
--------------------------------
1) Native Image (./tvdformat)
  44.91s user 1.82s system 305% cpu 15.317 total
  45.12s user 1.69s system 305% cpu 15.304 total
  45.85s user 1.61s system 309% cpu 15.344 total
2) GraalVM (java -jar ./tvdformat.jar)
  77.32s user 4.22s system 365% cpu 22.297 total
  77.59s user 4.32s system 371% cpu 22.044 total
  82.49s user 4.77s system 372% cpu 23.451 total
3) ARM JDK (java -jar ./tvdformat.jar)
  17.47s user 0.62s system 244% cpu 7.385 total
  17.02s user 0.57s system 242% cpu 7.262 total
  17.84s user 0.66s system 246% cpu 7.495 total


I used the result of the second execution per variant to create the chart.

Scenario 2 - Formatting 56 Files - Native Image vs Executable JAR

The native image is about 30% faster and uses about 40% less CPU resources. Rosetta 2 leads to an overhead of about factor 4.7 from a CPU usage perspective.

Conclusion

Wow, just wow, when I look at the performance and resource consumption figures of a native image. Startup times are really amazing and I was surprised by the results when formatting 56 files. The native image only consumes about half of the CPU resources compared to the executable JAR variant. This means lower operating costs while improving the end-user experience.

The price for a native image is higher development costs and the risk of runtime errors that would not occur in traditional Java environments. You definitely need to adjust your testing strategies to mitigate that risk. This is mandatory, not an option.

In this blog post, I eliminated all known runtime errors after four builds. In reality, it took much longer. I spent a lot of time hunting down the reasons for different behaviours between the executable JAR and the native image. This might improve once the debugger becomes available for the community edition. But even then you have to build a new image after a change. Only then you can start debugging. This results in long feedback loops. It does not matter whether you instrument your code or use a debugger.

This technology is fairly new. I’m sure that the tooling will improve over time. In the meantime, I’d prefer to use native images only for simple artefacts.

The post GraalVM Native Image – First Impressions appeared first on Philipp Salvisberg's Blog.

]]>