In my previous blog post, I’ve shown how you can deploy an npm module from a URL and a custom ESM module from a local file into a remote Oracle Database 23c using JavaScript and SQLcl. This works well. However, for two MLE modules, I had to write 22 lines of code with duplications. I do not like that. I have therefore developed a small SQLcl custom command that greatly simplifies the installation of MLE modules in the Oracle Database.

Installing the mle Custom Command

Start SQLcl and run the following command to install the custom command mle in the current session.

script https://raw.githubusercontent.com/PhilippSalvisberg/mle-sqlcl/main/mle.js register

The SQLcl script command can read a JavaScript file from the local file system or from a URL. If you feel uncomfortable running JavaScript files directly from a URL, you can have a look at the GitHub repo first and download and run a chosen version of the script from the local file system.

The register subcommand registers the mle script as an SQLcl command. However, this registration is not permanent. It will be lost after closing SQLcl. To make the custom command available in every new SQLcl session add the command above to your login.sql or startup.sql. SQLcl executes these files on start-up or after establishing a new connection. Just make sure that you have configured the SQLPATH environment variable accordingly.

Providing Help

Now you can run the mle command like this:

mle

Without parameters, an error message is displayed along with information on how to use this command.

Installing Validator

Now we know the syntax to install an MLE module from a URL. However, what’s the URL for an npm module? We use the free OpenSource CDN jsDelivr for that. They provide a service returning an npm module as an ECMAScript module. The result can be used in a browser and also in an Oracle Database. The URL looks like this:

https://esm.run/npm-package-name@npm-package-version

We want to install the current version 13.11.0 of the npm module validator. Based on the information provided above our SQLcl command for that looks like this:

mle install validator_mod https://esm.run/validator@13.11.0 13.11.0

And here is the result in SQLcl :

Please note that the response message by SQLcl 23.3 is not 100% correct for MLE modules. However, our module is installed correctly. We verify that later.

Maybe you’d like to know what the SQL statement looks like to install this module. The last statement is still in SQLcl’s buffer. Therefore we can type l followed by enter to see the content of the buffer.

SQLcl's buffer

The first line contains the start of the SQL statement. Lines 2 to 7 are comments generated by jsDelivr. Line 8 contains the complete module. Yes, as a single line. The code is minified. All unnecessary whitespace is gone and internally used identifiers are shortened to save space. On line 9 we would see a JavaScript comment with a pointer to a map file that points to the original, nicely formatted source code. This is interesting when debugging in other environments. It’s currently not used in the database.

Querying MLE Modules

The following SQL statement shows some data regarding the previously deployed MLE module:

set sqlformat ansiconsole
select module_name, version, language_name, length(module)
  from user_mle_modules;

The result in SQLcl looks like this:

We see the version of the module and also the size in bytes of the blob column where the module is stored. It is one byte larger than the result of the URL since It contains a final line feed character due to the way we built the SQL statement.

IMO it is helpful to provide the version of the external ESM as long as there is no proper package registry within the Oracle Database.

Verifying Installation

Let’s write a call specification in order to verify the successful installation of the validator module.

create or replace function is_email(
   in_email in varchar2
) return boolean as mle module validator_mod signature 'default.isEmail(string)';
/

Now we can run the following query to verify e-mail addresses and the installation of the validator module:

select is_email('jane.doe@example.org') as jane_doe,
       is_email('john.doe@example') as john_doe;

In SQLcl 23.3 the result looks like this (boolean values as 1 and 0):

And in SQL*Plus 23.3 the result looks like this (boolean values as TRUE and FALSE):

Installing More Modules

Let’s install some other modules I find useful. You find them on npm if you want to know what they do.

mle install sentiment_mod https://esm.run/sentiment@5.0.2 5.0.22
mle install jimp_mod https://esm.run/jimp@0.22.10/browser/lib/jimp.js 0.22.10
mle install sql_assert_mod https://esm.run/sql-assert@1.0.3 1.0.3
mle install lodash_mod https://esm.run/lodash@4.17.21 4.17.21
mle install js_yaml_mod https://esm.run/js-yaml@4.1.0 4.1.0
mle install minimist_mod https://esm.run/minimist@1.2.8 1.2.8
mle install typeorm_mod https://esm.run/typeorm@0.3.17 0.3.17

Installing all modules is just a matter of a few seconds.

Conclusion

Using the SQLcl custom mle command to install ECMAScript modules from a file or from a URL is not only easy and fast, but it is also an excellent way to provide tested functionality within the database.

Technically it should be possible to generate the PL/SQL call specifications based on the information that is provided for IDEs to better support code completion (type definitions). I hope that Oracle will provide such a feature in one of the coming releases of SQLcl or as part of a dedicated MLE tool (similar to dbjs which was part of the experimental version of the MLE back in 2017). Even if I do not want to provide access to all underlying functionality of an MLE module within the database or provide the functionality with the same signature, a PL/SQL package with all call specifications would simplify the work for a wrapper PL/SQL package that exposes just the relevant subset in a suitable way for the use in PL/SQL or SQL.

The post Installing MLE Modules in the Oracle Database appeared first on Philipp Salvisberg's Blog.

SQLcl 20.3.0 was released on October 29, 2020. It’s the first time I remember that we have a SQLcl version without the corresponding SQL Developer version. This is a pity because this SQLcl version also contains formatter fixes. And formatting code is something I do more often in the full-blown IDE than in the stripped-down command line interface. In this blog post, I show you how you can patch your SQL Developer installation and make the formatter fixes available there as well.

Disclaimer

What I do here is an experiment and most probably not really legal. It’s nothing I suggest you should do in your production environment. If you feel uneasy, don’t do it. What you do might destroy your SQL Developer installation.

Prerequisites

For this experiment you need

• SQLcl 20.3.0
• SQL Developer 20.2.0
• JDK8 or JDK11

These software components are all installed on your computer. I do not explain in this blog post how to do it.

Solution Approach

SQLcl and SQL Developer provide a dbtools-common.jar file. This file contains the formatter (besides other things). We copy this file from the SQLcl installation to the SQL Developer installation.

That would be all if SQL Developer would load the Guava library before opening a connection. But it does not because it was not required for version 20.2 of dbtools-common.jar. Therefore we put the VersionTracker.class and the SetServerOutput.class (which do not use Guava) from the original SQL Developer installation into the dbtools-common.jar file.

Another option would be to embed the classes from the guava.jar.

Step by Step Instruction

Step 1 – Quit SQL Developer

We are going to patch SQL Developer. This is not possible on Windows if SQL Developer is running. On other OS this might have strange effects. Therefore quit SQL Developer.

Step 2 – Rename SQLDev’s dbtools-common.jar

Find the dbtools-common.jar in your SQL Developer installation. In my case the file is in this directory: /Applications/SQLDeveloper20.2.0.app/Contents/Resources/sqldeveloper/sqldeveloper/lib. Rename this file to dbtools-common.original.jar.

cd /Applications/SQLDeveloper20.2.0.app/Contents/Resources/sqldeveloper/sqldeveloper/lib
mv dbtools-common.jar dbtools-common.original.jar

cd C:\app\sqldeveloper20.2.0\sqldeveloper\lib
ren dbtools-common.jar dbtools-common.original.jar

Step 3 – Copy SQLcl’s dbtools-common.jar

Find the dbtools-common.jar in your SQLcl installation. In my case, the file is in this directory: /usr/local/bin/sqlcl/lib. Copy the file to the SQLDev’s directory (where the dbtools-common.original.jar is located).

cd /usr/local/bin/sqlcl/lib
cp dbtools-common.jar /Applications/SQLDeveloper20.2.0.app/Contents/Resources/sqldeveloper/sqldeveloper/lib

cd C:\app\sqlcl\lib
copy dbtools-common.jar C:\app\sqldeveloper20.2.0\sqldeveloper\lib

Step 4 – Patch dbtools-common.jar

Open a terminal window and change to the directory of the dbtools-common.original.jar file. Run there the following commands to create a patched version of dbtools-common.jar:

cd /Applications/SQLDeveloper20.2.0.app/Contents/Resources/sqldeveloper/sqldeveloper/lib
jar -xvf dbtools-common.original.jar oracle/dbtools/db/VersionTracker.class
jar -xvf dbtools-common.original.jar oracle/dbtools/raptor/newscriptrunner/commands/SetServerOutput.class
jar -u0vMf dbtools-common.jar oracle/dbtools
rm -rf oracle

cd C:\app\sqldeveloper20.2.0\sqldeveloper\lib
jar -xvf dbtools-common.original.jar oracle/dbtools/db/VersionTracker.class
jar -xvf dbtools-common.original.jar oracle/dbtools/raptor/newscriptrunner/commands/SetServerOutput.class
jar -u0vMf dbtools-common.jar oracle/dbtools
rmdir /s /q oracle

Here is some explanation:

• First, we change to the directory where the files dbtools-common.original.jar (version 20.2.0) and dbtools-common.jar (version 20.3.0)  are stored. In your case, the directory name may differ.
• On the second and third line, we extract the VersionTracker.class and the SetServerOutput.class from the dbtools-common.original.jar file. The classes are stored in a newly created directory oracle.
• On the fourth line, we copy the previously extracted classes into the dbtools-common.jar file.
• And on the last line, we remove the oracle directory and its subdirectories.

Formatter Improvements

An example is the best to show the difference between version 20.2 and 20.3. I use the default formatter settings with just one change. “No breaks” for “Line Breaks On Boolean connectors”.

And here are the results after formatting the code once.

There are two significant improvements to the formatter. Both are related to comment and whitespace handling and therefore are independent of an Arbori program.

• The line of code after a comment is no longer indented additionally.
• The line break after a single-line comment is no longer lost. As a result, the formatter will no longer comment out code.

While the first bug leads to badly formatted code, the second bug is really nasty. It breaks your code. This happens when you use single-line comments on consecutive lines. In most cases, this will lead to compile errors. However, I showed that the resulting code may be syntactically correct and a wrong formatting result could go unnoticed.

Updated on 2020-11-03, fixed typo in jar file names (db-common... to dbtools-common...).

Updated on 2020-11-05, added SetServerOutput.class to the patched jar file. Otherwise, no SQL can be processed in a worksheet. Mentioned an alternative option to embed the classes from the guava.jar. Renamed chapter “Overview” to “Solution Approach”.

The post Patching SQL Developer 20.2 with SQLcl’s Formatter appeared first on Philipp Salvisberg's Blog.

Everything Changes. Our Trivadis SQL & PL/SQL Coding Guidelines are no exceptions. We plan to change rule #1 of our coding styles. From “Keywords are written uppercase, names are written in lowercase.”  to “Keywords and names are written in lowercase.“. We have 103 Markdown files and most of them contain several SQL code blocks complying to our current (old) rule #1. Should we change these files manually? Nah, this is boring and error-prone. It’s a perfect case to automate it and to show how you can format SQL code blocks in Markdown with SQLcl.

Tools

For this task, we use SQLcl 20.2.0. If you work with Oracle databases, you have most likely already installed it.

SQLcl is basically SQL*Plus on steroids. One of the most underestimated features of SQLcl is the ability to execute JavaScript and provide them as custom SQLcl commands (read Erik van Roon‘s excellent blog post to learn more about it). We use the custom command tvdformat. To install it, save format.js locally in a folder of your choice. Then start SQLcl (no connection required), go to the folder where you’ve saved format.js and run script format.js -r. This will register the command tvdformat. You get the usage help, when you enter the command without arguments.

Formatting a Single Markdown File

Let’s create a simple Markdown file to see how the formatter behaves.

## SQL to be formated

``` sql
SELECT * FROM EMP JOIN DEPT ON EMP.DEPTNO = DEPT.DEPTNO;
```

## SQL to be ignored

```
SELECT * FROM EMP JOIN DEPT ON EMP.DEPTNO = DEPT.DEPTNO;
```

## JavaScript to be ignored

``` js
var foo = function (bar) {
  return bar++;
};
```

Save the content in a file named example.md. And then run tvdformat example.md. This will format this file with default settings. Default means with the embedded advanced settings (xml) and the default custom settings (arbori).

The result should look like this:

## SQL to be formated
 
``` sql
SELECT *
  FROM EMP
  JOIN DEPT
ON EMP.DEPTNO = DEPT.DEPTNO;
```
 
## SQL to be ignored
 
```
SELECT * FROM EMP JOIN DEPT ON EMP.DEPTNO = DEPT.DEPTNO;
```
 
## JavaScript to be ignored
 
``` js
var foo = function (bar) {
  return bar++;
};
```

As you see only the first SQL statement is formatted. The other code blocks are left as is. Only code blocks with sql syntax highlighting are formatted.

The indentation of line 7 is wrong. It’s an issue of the default Arbori program. It’s addressed in trivadis_custom_format.arbori. However,  we do not want to format the code blocks anyway. We just want to change the keywords and identifiers to lowercase.

Changing Keywords and Identifiers to Lowercase

You can export the advanced format settings in SQL Developer. When you look at the options in the resulting XML file, the first option is adjustCaseOnly. This option cannot be set in the Advanced Format preferences. It’s set to false by default. When changed to true the formatter still executes some part of the Arbori program, but basically skips all actions that deal with whitespace before a node. Knowing that we can create the following options.xml file:

<options>
    <adjustCaseOnly>true</adjustCaseOnly>
    <idCase>oracle.dbtools.app.Format.Case.lower</idCase>
    <kwCase>oracle.dbtools.app.Format.Case.lower</kwCase>
</options>

Let’s reset the content of example.md to the unformatted one. And then run tvdformat example.md xml=options.xml.

Now, the result should look like this:

## SQL to be formated

``` sql
select * from emp join dept on emp.deptno = dept.deptno;
```

## SQL to be ignored

```
SELECT * FROM EMP JOIN DEPT ON EMP.DEPTNO = DEPT.DEPTNO;
```

## JavaScript to be ignored

``` js
var foo = function (bar) {
  return bar++;
};
```

As before, only the first code block changed. In this case, everything is in lowercase. However, the processing is more complicated behind the scenes. For example: comments, strings, and quoted identifiers are left untouched. So, it’s more than just a simple .toLowerCase() call and for sure worth using Oracle’s formatter for this task.

Is it Safe to Change the Case in PL/SQL & SQL?

PL/SQL & SQL are case-insensitive languages. So you might be tempted to answer this question with “Yes”. But it is not that easy. For keywords, it’s 100% true. However, this is not true for identifiers. Roger Troller was the first who showed me examples of unquoted, case-sensitive identifiers in SQL. One is documented here.  For example, if you use JSON columns the items in the JSON document are case-sensitive. Changing the case will break the code. That’s bad. This is also the reason, why we do not change the case of identifiers in our formatter configuration.

Therefore, be careful, if you change the case of identifiers. This might break your code. Depending on your test coverage you might detect this problem very late, because the program might still compile, but not produce the expected results anymore (as in the mentioned example).

Bulk Processing

In our case, we know that we do not have JSON-based code snippets in our Markdown files. Therefore it is safe to change the case of identifiers in all files.

To process all files in the docs directory including all subdirectories I run tvdformat docs xml=options.xml arbori=default. I pass the arbori option only to avoid the warning message.

In this case, the code is based on a Git repository. Therefore I can browse through the changes before committing them. Here’s an excerpt of the g-1050.md file.

You see that the original whitespace is preserved. Only keywords and identifiers are changed to lowercase. The string ‘AD_PERS’ is still in uppercase. This looks good. Ready to be checked in.

Recommendations

The current formatter settings are probably not good enough for all code. There are for sure some cases where the original code base is formatted so badly, that an imperfect formatting configuration leads to a huge improvement. But generally, this is not good enough. You can use the formatter when writing code. That includes changing existing code when the current formatting style makes it difficult to read. You can always select a portion of code (a subquery, a function, etc.), format it and then change the things you don’t like. It’s easy to undo the changes in the IDE. This is also possible if you apply the formatter for a large number of files, especially if you use a version control system such as Git. It is simple to undo everything. However, when you change hundreds of files you will easily overlook some uglified code.

For bulk processing, changing the case of keywords is safe. Changing the case of identifiers is possible. But be careful, if you are using case-sensitive SQL, this will break your code.

Whatever you do, make sure you keep the version before applying the formatter. And do not forget to test and review the result.

The post Formatting SQL Code Blocks in Markdown Files appeared first on Philipp Salvisberg's Blog.