Formatting Code With SQL Developer

blank

Introduction

I started using SQL Developer in 2013. Back then version 4.0 was the latest and greatest. But the capabilities of the formatter were disappointing. In 2017 Oracle released version 4.2 with a new formatter and has been improving it ever since. Version 19.2 brought us dynamic JavaScript actions within the parse-tree query language Arbori. And now I must admit that I’m really impressed with the formatting capabilities of the latest versions of SQL Developer. Arbori is a hidden gem.

In this blog post I explain how the formatter works and how the output can be tweaked using two simple SQL queries.

If you only want to activate the coding styles suggested by the Trivadis PL/SQL & SQL Coding Guidelines, install the settings as described here.

How Does Formatting Work?

Formatting is all about adding (or removing) whitespaces (line breaks, spaces or tabs) between significant tokens. That sounds easy. Well, it’s not. Because the formatting requirements are very different. Ultimately, it’s all about beautifying the code. And almost every developer has his own views on what makes code look good. Furthermore, it is technically demanding to provide a tool suite that is able to handle different coding styles via configuration.

The following figure illustrates the formatting process in SQL Developer.

blank

I will explain each step and component in the next chapters.

Please note that these are conceptual components, the actual implementation might look different.

1. Parser

The parser reads the unformatted plain SQL or PL/SQL input and generates a parse-tree. The parse-tree is a hierarchical representation of the significant tokens of the input. In other words, there are neither whitespaces nor comments in a parse-tree.

Each node in the parse-tree includes the start and end position within the plain SQL input.

2. Formatter

The formatter needs the parse-tree and the code formatting configuration as input.

SQL Developer stores the configuration in the preferences.

  • Under Code Editor -> Format -> Advanced Format for configuration properties such as Line breaks on comma (after, before, none).
  • And under Code Editor -> Format -> Advanced Format -> Custom Format for the Arbori program used to handle whitespaces.

2.1 Provided Java Callback Functions

The formatter provides the following Java callback functions (in the order how they are expected to be called):

  • indentedNodes1
  • indentedNodes2
  • skipWhiteSpaceBeforeNode
  • skipWhiteSpaceAfterNode
  • identifiers
  • extraBrkBefore
  • extraBrkAfter
  • brkX2
  • rightAlignments
  • paddedIdsInScope
  • incrementalAlignments
  • pairwiseAlignments
  • ignoreLineBreaksBeforeNode
  • ignoreLineBreaksAfterNode
  • dontFormatNode

Each callback functions gets the parameters target (the parse-tree) and tuple (node to be processed). As an Arbori developer you do not have to care about how to populate these parameters. It’s done automatically. target is a global variable and tuple is the result row of an Arbori query. Basically, you only need to query the nodes and call the callback functions. The position in an Arbori program defines the execution order.

These provided Java callback functions have two issues.

First of all, you don’t know what they do. Granted, there are some comments in the provided Arbori program, and also a description in the SQL Developer Users Guide, but this will only give you a rough idea. For example, it leaves you in the dark why indentedNodes has two callback functions and both must be called.

Second, you cannot process selected nodes differently. You must write an enhancement request so that the SQL Developer team can provide the necessary callback functionality in a future release. This is cumbersome.

2.2 JavaScript Callback Functions

Thankfully, the SQL Developer team has added a JavaScript callback feature in version 19.2. This allows you to embed callback functions directly into your Arbori program. Now you can really add and remove whitespaces wherever you want. The global variable struct gives you access to the instance of the formatter and the configuration properties. As a result, you can manage the whitespaces before a position of a node through the methods getNewline and putNewline.

2.3 The Result

Basically, the result of this process is a list of whitespaces per position.

3. Serializer

The serializer loops through the leaf nodes of the parse-tree. It retrieves the leading whitespaces for a node’s start position and extracts the token text from the pure SQL input using the node’s start and end position. And then the serializer writes the whitespaces and the token text to the final result. The formatted SQL.

In fact, the process is actually a bit more complicated. It adds whitespaces to mandatory nodes, for instance.

Moreover, the serializer performs some “formatting” without Arbori. For example, it converts the case of identifiers and keywords according to the configuration (properties). Therefore, it is not possible to change the case of a token with an Arbori program. It might be possible by configuring a custom Java formatter class, but that’s another story.

Example Using Provided Java Callback Function

Setup

For this example I use the Advanced Format according  the trivadis_advanced_format.xml file. Here’s a screenshot of the configuration settings of my SQL Developer 19.4.0 installation:

blank

The default is used for the Custom Format.

Default Formatter Result

The result looks good, beside the missing indentation on line 6.

Expected Formatter Result

What we expect is this:

The ON keyword right-aligned as SELECT, FROM, LEFT and ORDER.

Code Outline

SQL Developer’s code outline is in fact a representation of the full parse-tree. Disable all filters to show all nodes.

blank

The highlighted information is important for the next step.

Arbori Editor

Type arbori in the search field and press enter as shown below:

blank

This will open the Arbori Editor. Type the following query in the editor window:

Press Run to display the query result:

blank

What have we done? We query the parse-tree (outline) for all ON nodes where the parent node is an on_using_condition. A node is represented as  [node). And a parent node is represented as [node^). A boolean AND is represented as &.  See these links for more information about the Arbori grammar.

Click on the query result cell [19,20) 'ON' to highlight the node in the Code Outline window and the corresponding text in the worksheet. You can do the same with the cell [19,27) on_using_condition.

Change in Arbori Program

Now open the Preferences  for Custom Format and search for the query named rightAlignments (it’s usually easier to change the Arbori program in separate editor). It looks like this:

blank

Here some explanation of the query:

  • The predicate :alignRight means that the option Right-Align Query Keywords must be checked (true).
  • We know the boolean AND & ,  the current node [node)  and the parent node [node^) from the previous query.
  • The parenthesis ( and ) are part of the boolean expression.
  • The |  is a boolean OR.
  • The -> at the end means the callback function named as the query (rightAlignments) is called for matching nodes.
  • -- is used for single-line comments as in SQL and PL/SQL.

We extend the query by the predicate | [node) 'ON' & [node^) on_using_condition  to right-align the ON token.

Here’s the amended query:

blank

Press OK to  save the preferences. Now, the query is formatted correctly.

Example Using JavaScript Callback Function

Default Formatter Result

We use the same setup as for the previous example.

The result does not look too bad. But the indentation feels wrong. Especially when I look at the missing indentation of  the ) on line 8. Therefore, I’d like to increase the indentation of the highlighted lines by 7.

Expected Formatter Result

What we expect is this:

Look at the indentation on line 8. ) matches now the indentation of EXISTS (.

Change in Arbori Program

The highlighted code block is already indented. Therefore we cannot use the same mechanism as previously. We want an additional indentation. We can achieve that with an additional query and a JavaScript callback function.

Add the following query at the end of the existing Arbori program in Custom Format of the Preferences:

Here are some explanation:

  • On line 3 and 4 the predicates are defined, for the subquery and the closing parenthesis ) of an exists_condition.
  • The JavaScript callback starts on line 6 and ends on line 33.
  • The current indentation of a node (position) is read on line 27 and updated on line 29.

Save the preferences to enable the new formatting rules. This is a reduced example. See the PL/SQL & SQL Formatter Settings repository on GitHub for a more complete Arbori program.

Summary

Arbori is the flux capacitor of SQL Developer’s Formatter. Arbori is what makes highly customized code formatting possible.

The Arbori Editor and Code Outline are very useful tools for developing code snippets for an Arbori program. However, it is not easy to get started with Arbori. The information in Vadim Tropashko’s blog is extensive, but it is a challenging and time-consuming read. For me, it was definitely worth it. I hope this blog post helps others to understand Arbori and its potential a bit better.

Any feedback is welcome. Regarding this blog post or the PL/SQL & SQL Formatter Settings on GitHub. Thank you.

6 Comments

  1. blank Vadim Tropashko says:

    I  have added your ON clause alignment and extra incremental_alignment conditions, thank you. Will revise and possibly other changes too.

    Here are some 20.1 rule enhancements (the release seems to be on hold):

    • Thank you, Vadim. I appreciate.

      As soon as SQL Developer 20.1 is out I’ll merge the changes into the trivadis_custom_format.arbori file on GibHub.

      I’ll be more than happy if the differences to the original Arbori file can be reduced. However, I’m not sure if my changes are compatible with/applicable to all combinations of the “Advanced Format” settings (I guess they are not). Basically, I tested only “my favorite” settings with a few variations (comma before/after and number of spaces of indentation).

      And talking about testing. That’s something I’ve done manually in SQL Developer. I have to figure out how to automate that.

      • Vadim helped me via e-mail to understand the formatting process better, showed me some Arbori tricks and how I could run formatting tests outside of SQL Developer (or SQLcl).

        Thank you very much, Vadim!

        I extended the repository by a test-only Maven project to run formatting tests with JUnit.

  2. blank Anand says:

    Hi Philipp
    Thanks for the details  about how to configure settings
    have question regarding

    DECODE

    is there a way i can configure the setting not to split like below.
    i see it is splitting because of <breaksProcArgs>true</breaksProcArgs>  in the format.xml
    but i do want the procedure inputs to be split but not for the decode

    desired 

    • Hi Anand,

      When I format the code with the default custom settings (Arbori program) in SQL Developer 19.4.0, the result for a complete SQL statement is the following:

      By importing the advanced format trivadis_advanced_format.xml and the custom format trivadis_custom_format.arbori the formatting result looks like this:

      That’s what you want, right? However, instead of adding an exception for DECODE  I add a line break only if named parameters are used. The next example shows that:

      For that I had to change the following in the Arbori program:

      If you really want to change the behavior for DECODE only, you could add a fix at the end of the Arbori program that eliminates all whitespaces of the function’s child nodes. Something like this:

      You can ask questions regarding the formatter behavior in the SQL Developer forum. The SQL Developer team and others are watching this space and are usually very responsive. And if it’s related to the Trivadis PL/SQL and SQL formatter settings, it’s the best to open an issue on GitHub. Thank you.

  3. blank Anand says:

    Thanks alot for your reply and direction on this.
    Sure will do that.

     

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.