SecureFiles is the default storage mechanism for LOBs starting with Oracle Database 12c, and Oracle strongly recommends SecureFiles for storing and managing LOBs, rather than BasicFiles. BasicFiles will be deprecated in a future release.

Besides the mentioned warning, there are also performance considerations that drive us toward moving to SecureFiles. We won't delve into those - this post will focus on how to migrate from BasicFile to SecureFile. Which is simple with relatively small amount of rows:

ALTER TABLE my_table MOVE LOB (lob_column) STORE AS SECUREFILE;

But.. this can take significant amount of time. In my case, it took 15 hours for 6TB of LOBs. And while this is running, the table is locked - users can't issue DMLs on it.

If we were on Enterprise Edition, we could think of ONLINE move option or even better, dbms_redefinition; but those are specifically Enterprise Edition features according to Licensing Guide.

So, here's an idea on approach somewhat similar to the use of dbms_redefinition which works on Standard Edition. Be warned though, this idea focuses solely on cloning the table data - everything else (explained at the end of this post) is up to the developers/DBAs and not covered by this post.

Preparation

First, we create a new, empty table, with the same columns as our BasicFile source table. We can use dbms_metadata.get_ddl to obtain original DDL and modify it to specify SecureFile instead of BasicFile.

So, thus, suppose that now we have two tables:

• my_basicfile with millions of rows
• my_securefile which is empty for now

You could also specify NOLOGGING on the new table - just don't forget that in most cases you'll also want to change this back to LOGGING and take appropriate backup before letting users use the new table. It should also go without saying that NOLOGGING operations won't propagate to your physical standby.

Anyway, for most of actions described after this point, it's best to execute them while there are no users online (perhaps using restricted session?).

This example requires that the BasicFiles table have a primary key. The same concept might work WITH ROWID clause for CREATE MATERIALIZED VIEW LOG, but this isn't the case in this example.

Initial Clone

Let's start by creating Materialized View and a MV Log:

CREATE MATERIALIZED VIEW LOG ON my_basicfile
    WITH PRIMARY KEY
    INCLUDING NEW VALUES;

CREATE MATERIALIZED VIEW my_securefile
	ON PREBUILT TABLE
	REFRESH FAST
	AS SELECT * FROM my_basicfile;

The log is used to track all changed rows (ins/upd/del) on the source table since its creation. We'll need it in the next step for fast refresh. This log must be created before dbms_mview.refresh() is called and before users are allowed to make changes to the my_basicfile.

Note that the MV log only stores the primary keys of changed rows - the actual LOB data is read from the source table during refresh.

The MATERIALIZED VIEW is created on prebuilt table, which is actually empty. So, we need to populate it:

begin
    dbms_mview.refresh(
        list => 'my_securefile',
        method => 'C', 
        atomic_refresh => FALSE);
end;

The method => 'C' indicates complete refresh (copying all rows from the my_basicfile source).

atomic_refresh=>FALSE doesn't seem needed at first glance, but it's there so that the underlying method uses /*+ APPEND */ hint when inserting the data - which makes this complete refresh a bit faster. It also uses TRUNCATE before insert this way, but that doesn't matter since my_securefile is empty at this stage.

This dbms_mview.refresh is still likely to take significant amount of time - but the users can be online while it is running; remember, any changes they make to the underlying my_basicfile is recorded in a materialized view log.

Under the hood, according to my tracing on 19c, following is executed:

LOCK TABLE "LOB_TEST"."MY_SECUREFILE" IN EXCLUSIVE MODE  NOWAIT;
/* MV_REFRESH (DEL) */ truncate table "LOB_TEST"."MY_SECUREFILE" purge snapshot log;
/* MV_REFRESH (INS) */INSERT /*+ BYPASS_RECURSIVE_CHECK APPEND SKIP_UNQ_UNUSABLE_IDX */ INTO "LOB_TEST"."MY_SECUREFILE"("COL_PK","COL_CLOB") SELECT "MY_BASICFILE"."COL_PK","MY_BASICFILE"."COL_CLOB" FROM "LOB_TEST"."MY_BASICFILE" "MY_BASICFILE";

Fast Refresh

After previously mentioned complete refresh finishes (this could be, say, the next evening), we only need to add changes that gathered in materialized view log since we ran the full refresh. Like this:

exec dbms_mview.refresh('my_securefile', method => 'F');

F stands for Fast refresh. This means, that Oracle will use the materialized view log, which is just a table (automatically populated by Oracle as changes occur) named mlog$_my_basicfile - you'll find list of all primary keys that were touched in there.

Under the hood, according to my sql tracing on 19c, it runs:

DELETE FROM "LOB_TEST"."MY_SECUREFILE" SNAP\( WHERE "COL_PK" IN (SELECT * FROM (SELECT MLOG\)."COL_PK" FROM "LOB_TEST"."MLOG\(_MY_BASICFILE" MLOG\) WHERE "SNAPTIME$$" > :1 AND ("DMLTYPE$$" != 'I')) AS OF SNAPSHOT(:B_SCN) );

/* MV_REFRESH (MRG) */ MERGE INTO "LOB_TEST"."MY_SECUREFILE" "SNA\(" USING (SELECT * FROM (SELECT CURRENT\)."COL_PK",CURRENT\(."COL_CLOB" FROM (SELECT "MY_BASICFILE"."COL_PK" "COL_PK","MY_BASICFILE"."COL_CLOB" "COL_CLOB" FROM "LOB_TEST"."MY_BASICFILE" "MY_BASICFILE") CURRENT\), (SELECT MLOG\(."COL_PK" FROM "LOB_TEST"."MLOG\)_MY_BASICFILE" MLOG$ WHERE "SNAPTIME$$" > :1 AND ("DMLTYPE$$" != 'D') GROUP BY MLOG\(."COL_PK") LOG\) WHERE CURRENT\(."COL_PK" = LOG\)."COL_PK") AS OF SNAPSHOT(:B_SCN) )"AV\(" ON ("SNA\)"."COL_PK" = "AV\("."COL_PK") WHEN MATCHED THEN UPDATE  SET "SNA\)"."COL_CLOB" = "AV\("."COL_CLOB" WHEN NOT MATCHED THEN INSERT  (SNA\)."COL_PK",SNA\(."COL_CLOB") VALUES (AV\)."COL_PK",AV$."COL_CLOB");

Cutover & Cleanup

For the final cutover, users should be offline again (restricted session?). Fast Refresh, as explained previously, should finish once again, just to make sure the mview log is all applied to the my_securefile.

So, once we're sure both our tables, my_securefile and my_basicfile have the same contents, we can drop the materialized views, the MV log and finally, also the my_basicfile table. Like so:

DROP MATERIALIZED VIEW my_securefile PRESERVE TABLE;
DROP MATERIALIZED VIEW LOG ON my_basicfile;
DROP TABLE my_basicfile PURGE;

but.. herein lies the problem, what if there were foreign keys referencing my_basicfile? What about permissions? Triggers? All those and more must be manually changed (this could mean altering a huge amount of database objects/tables), so that it points to our new table before we're ready to make the final rename:

ALTER TABLE my_securefile RENAME TO my_basicfile;

Possible Alternative (adding a column)

There is an alternative idea - what if, instead of creating a new table and copying everything over, we'd simply add another column? e.g.:

alter table my_basicfile
    add (col_securelob clob)
    lob (col_securelob) store as securefile;

then, simply update new column to the value of the old. There is no free lunch though - this approach might be much lighter regarding metadata (permissions, foreign keys, etc), but...

You'd need to consider the following challenges:

• Row Lock Contention
• Undo & Redo Generation

Those challenges can be addressed to certain extent by:

• updating in chunks
• setting nologging (temporarily) to the new lob column (e.g. store as securefile (nocache nologging))
• creating a trigger to update the new column when the old one is updated/inserted

A quick and naive comparison of this alternative on a slow demo server with generated demo data proved to be about 7x slower than a MV approach. YMMV, of course.

Conclusion

There are many (other) methods on how to migrate LOBs from basicfile to securefile - a simple blog post such as this one cannot possibly address all of them; This is why I focused on one specific approach which I found specifically interesting. It is by no means "the best" approach - it's simply one of the options.

I've tested this approach on 19c SE and 26ai EE.

Most would first think of the cost of process creation, memory structures allocation and initialization etc. Which is all true, except that there is another important thing that goes on - and it has the potential to affect all the other sessions (those already logged in) - the REDO!

This post will focus on what is going on with REDO at the time of new user login.

The Test Case

The original story was that "the database is 'slow' because of a lot of logins!". And I thought "that sounds unlikely". So I made a quick test case - a program which takes three parameters:

• connect string (e.g. user/pass@host:1521/service )

• number of minutes to run the test

• number of parallel threads in which to run the test, but we'll always use only 1 in this blog post.

Each thread does only the following:

• connect

• select * from dual;

• disconnect

... as many times as possible. The code of this test case is available as GitHub Gist.

Note that this test case was run on fairly "slow" vm; which is intentional - you can spot performance issues faster on slower machines :)

ASH

Here's the 1 hour ASH AAS (average active sessions per minute) for the period where the only load on the database was the given test case:

Notice the "orange" and “blue” colors? Those are log file parallel write andlog file sync in this case. So, what was being written to redo?

(btw, the ASH images are screenshots from the free APPM tool).

Oracle Log Miner

To find out what was written to redo logs I decided to "craft" an archivelog by:

• alter system archive log current

• run the test for 1 minute

• alter system archive log current

My test case said:

Threads: 1
Total time: 60.03 seconds
Total successful connections: 770
Total errors: 0                
Total connections per second: 12.83

Now let's see what LogMiner has to say regarding operations on SYS.USER$ segment:

with
   user_ops as (
   select *
      from v$logmnr_contents
      where seg_owner='SYS' and seg_name='USER$'   
   )
select c.operation, count(*) 
   from user_ops u
   join v$logmnr_contents c on c.xid=u.xid
   group by c.operation;

OPERATION         COUNT(*)
--------------- ----------
START                  770
UNSUPPORTED            770
COMMIT                 770

The query first finds all the changes on SYS.USER$ segment and then joins all other operations from the same transactions as the changes (XID is the transaction id). This means we can confirm that the START, UNSUPPORTED, and COMMIT operations are part of the same transaction triggered by each login, not unrelated activity.

Notice how there is exactly as many transactions on SYS.USER$ as there are logins (770)? So, according to this, each login does "update" on SYS.USER$ and a COMMIT. In our "connection storm", we managed to perform 12 commits per second just by logging in 12 times per second. And those numbers can get even higher when users are logging in concurrently.

Unsupported Operation

Official documentation says regarding such operations: "Change was caused by operations not currently supported by LogMiner", so, we need to resort to redo dump next:

ALTER SYSTEM DUMP LOGFILE '/path/to/arch1_112_1223914063.dbf' SCN MIN 6365975 SCN MAX 6365975;

26ai

All 770 of redo records contain this change on SYS.USER$:

CHANGE #3 ... OP:11.5 ...
ncol: 30 nnew: 1 size: 0
col 18: [ 1]  80

Column 18, if I'm not mistaken is SPARE1 column of SYS.USER$ table. Value 0x80 is Oracle's internal representation of NUMBER 0. In DBA_USERS, this column is used with bitand() to check for many different flags.

But - of those 770 redo records, 122 of them, besides the described change, also contain an update of column 23, which is SPARE6 column of SYS.USER$, which is of datatype DATE and refers to DBA_USERS.LAST_LOGIN:

ncol: 30 nnew: 2 size: 0
col 18: [ 1]  80
col 23: [ 7]  78 7e 02 10 0b 35 0f

This tells us, that even though my test case was doing about 12 logins per second, the LAST_LOGIN column was updated two times per second.

19c

On 19.20.0, the situation is pretty similar for each login:

CHANGE #3 ... OP:11.19 ...
Array Update of 1 rows: 
ncol: 28 nnew: 1 size: 0
col 23: [ 7]  78 7e 02 10 10 26 1e

So, instead of changing column 18, it either applies an empty change array or updates the last login date. Regardless, it still makes and commits a transaction for each login, just like 26ai does.

Older versions

I didn't test older versions, but it might be interesting to know that according DBA_USERS 12c documentation, the DBA_USERS.LAST_LOGIN column was introduced in version 12c.

_disable_last_successful_login_time

This is the undocumented parameter which greatly reduces redo generation on logins. However, as it is with all undocumented parameters, you should only use them in accordance with Oracle Support guidance.

Here's the same test case with _disable_last_successful_login_time set:

select count(*) as update_count
   from v$logmnr_contents
   where seg_owner='SYS' and seg_name='USER$';

select count(*) as commit_count
   from v$logmnr_contents
   where operation='COMMIT';

UPDATE_COUNT
------------
           0


COMMIT_COUNT
------------
           9

So, truly no commit per login anymore. There are 9 commits, yes, but those were unrelated to logins. This not only disables last successful login time update, but also the update on SPARE1 - after enabling the parameter, no USER$ transaction per login was observed in LogMiner output.

Conclusion

In tested 19c/26ai environments with default settings, each regular user login resulted in a recursive transaction and commit. During a connection storm, these login-driven commits add to the LGWR workload alongside commits from actual application activity. So the hidden cost isn't just the login itself - it's the commit latency impact on every other session. This kind of issue is quite rare, but I find it valuable to know that, by default, there is a transaction after each regular user login.

Rather than needing deep knowledge of ASM's internal structures, asmfs can rely on the x$kffxp view, which exposes nearly all the metadata required. In this post, I'll walk through how asmfs leverages this view to read ASM files.

Disk Groups and Allocation Units (AU)

Here are the basics: an ASM diskgroup is made of one or more disks. Space for each file is allocated in uniformly sized "chunks" known as extents. The same extent can be written to one (aka external redundancy), two (normal redundancy) or three (high redundancy) different disks. So, for example, in a high redundancy diskgroup, each extent needs 3 allocation units (AU).

Each ASM diskgroup can have different AU size, but strictly all allocation units (AU) have the same size within one diskgroup. You can define your preferred AU size (1, 2, 4, 8, 16, 32 or 64 MB) when you create a diskgroup.

We can figure out our current AU size and ASM redundancy level (v$asm_diskgroup.type) by querying v$asm_diskgroup like this:

SQL> select group_number, name, type, allocation_unit_size / 1024 / 1024 as au_size_mb from v$asm_diskgroup;

GROUP_NUMBER NAME     TYPE     AU_SIZE_MB
------------ -------- -------- ----------
           1 DATA     NORMAL            4

SQL> select disk_number, path from v$asm_disk where group_number=1;

DISK_NUMBER PATH
----------- ---------------
          0 AFD:DATA1
          1 AFD:DATA2
          2 AFD:DATA3

If you're using AFD then you'll also need to resolve AFD labels to actual block devices like this:

$ afdtool -getdevlist
--------------------------------------------------------------------------------
Label                     Path
================================================================================
DATA1                     /dev/sdd
DATA2                     /dev/sdb
DATA3                     /dev/sde

File Number

In order to obtain the list of all allocation units (AU) that belong to a given file, we need to obtain the file_number of the file we want to read. It can be obtained by simply querying v$asm_alias like this:

SQL> select name, file_number from v$asm_alias where name = 'SYSTEM.261.1206729191' and group_number=1;

NAME                           FILE_NUMBER
------------------------------ -----------
SYSTEM.261.1206729191                  261

x$kffxp (Extent Map)

Now the fun part:

SQL> SELECT
    x.disk_kffxp AS disk_number,
    x.au_kffxp AS allocation_unit
FROM x$kffxp x  
WHERE x.group_kffxp = 1        /* :group_number */
    AND x.number_kffxp = 261   /* :file_number */
    AND x.lxn_kffxp = 0        /* :mirror */
ORDER BY x.xnum_kffxp /* extent number */;

DISK_NUMBER ALLOCATION_UNIT
----------- ---------------
          2             148
          0             147
          1             152
...

By using first two where conditions (:group_number and :file_number) we select only those allocation units that belong to our file. But because our example diskgroup has normal redundancy, we get two allocation units for each extent. So, we filter by :mirror, which can be:

• 0 for primary copy
• 1 for secondary copy (available in normal and high redundancy diskgroups)
• 2 for tertiary copy (available in high redundancy diskgroups)

In asmfs, you can select which :mirror copy you want to use by specifying --mirror parameter to asmfs command (e.g. asmfs --mirror 1 for always reading only from secondary copy).

Order is also important, xnum_kffxp is extent number. First extent starts with 0, second one is 1 etc. A constant value of 2147483648 refers to the triple-mirrored file metadata.

So, how can we use disk number (x.disk_kffxp) and allocation unit number x.au_kffxp?

Running dd

Based on the example three rows returned by our example query, we can run three dd commands like this:

dd if=/dev/sde bs=4194304 skip=148 count=1 >> /tmp/SYSTEM.DBF
dd if=/dev/sdd bs=4194304 skip=147 count=1 >> /tmp/SYSTEM.DBF
dd if=/dev/sdb bs=4194304 skip=152 count=1 >> /tmp/SYSTEM.DBF
...

Let's explain the first command in detail:

• if=/dev/sde because DISK_NUMBER=2 from x$kffxp refers to PATH=AFD:DATA3 (according to v$asm_disk) which refers to /dev/sde (according to afdtool).
• bs=4194304 because AU size is 4 MB (according to v$asm_diskgroup.allocation_unit_size)
• skip=148 because allocation unit from x$kffxp.au_kffxp is 148
• count=1 because we're reading only one AU at a time

btw, if you're running AFD, you can only do such dd commands as root because one of the AFD features is to deny non-oracle non-root software direct access to block devices.

File size

Note that the file we've created by running all those dd commands must be multiple of AU size (4 MB in our example). But actual Oracle files (e.g. datafiles) can be any multiple of any valid block size (e.g. 8 KB). In other words, there is a high chance that our file contains garbage at the end of file, which we can get rid of by running truncate:

truncate -s 3659538432 /tmp/SYSTEM.DBF

The number of bytes to which we need to truncate is obtained simply from v$asm_file:

SQL> select bytes from v$asm_file where file_number=261;

     BYTES
----------
3659538432

File Headers

By using the approach described above, you'll get file contents as it is written in the ASM. The problem is that if you copy a file using, say rman or asmcmd, then md5sum will be different than what you'll get from described dd & truncate commands. The difference comes from first block only. Not really the complete first block, but just two 4-byte regions:

• The 4 bytes from 0x20 to 0x23 seems to be the magic constant, which is different for files in ASM and for files in a local filesystem. For files in local filesystem, this always seems to be the same magic value 0x000081a0 in little-endian format.
• The 4 bytes from 0x10 to 0x13 seems to be the checksum. Due to the way this checksum is calculated, we can simply read the existing checksum and XOR it with the previously mentioned magic value.

For more details, feel free to read the function fix_header_block in file fuse.rs.

Not every file type follows this rule, but most do - datafiles, tempfiles, and archivelogs all behave this way. I stumbled upon this magic constant by doing a simple byte-by-byte comparison of files and noticed it kept showing up with the same value.

Since we're deep in undocumented territory here, there's always a chance I've got something slightly wrong. But so far, it seems to work! :)

Getting started with ASMFS

Now that you understand how it works, feel free to grab a ready-made asmfs.rpm files compatible with Oracle Linux versions 10, 9, 8 from asmfs releases. Installation instructions and examples are all in the README.md of the github project.

If you run into issues or discover edge cases, feel free to open an issue on GitHub.

References

A lot of useful additional information regarding ASM Metadata and Internals is also available on twiki.cern.ch.

SQL> exec utl_recomp.recomp_serial('HR');
BEGIN utl_recomp.recomp_serial('HR'); END;

*
ERROR at line 1:
ORA-20000: Unable to gather statistics concurrently: the job_queue_processes parameter is less than 4

How does this make sense—serial recompilation wants to do concurrent statistics gathering? Also, the UTL_RECOMP documentation does not mention anything regarding statistics (I tried searching for the 'stat' keyword in this doc). So, what and why is it being gathered?

A quick spoiler, this happens if you set both of these:

alter system set job_queue_processes=0;
exec dbms_stats.set_global_prefs('concurrent', 'true');

Changing either one will make utl_recomp.recomp_serial() succeed.

Also, perhaps worth mentioning, this test case can be reproduced even if the example HR schema is empty.

Stats Gathering

This is the full stack trace:

SQL> exec utl_recomp.recomp_serial('HR');
BEGIN utl_recomp.recomp_serial('HR'); END;

*
ERROR at line 1:
ORA-20000: Unable to gather statistics concurrently: the job_queue_processes
parameter is less than 4
ORA-06512: at "SYS.UTL_RECOMP", line 927
ORA-06512: at "SYS.DBMS_STATS", line 40799
ORA-06512: at "SYS.DBMS_STATS", line 5065
ORA-06512: at "SYS.DBMS_STATS", line 40725
ORA-06512: at "SYS.UTL_RECOMP", line 260
ORA-06512: at "SYS.UTL_RECOMP", line 830
ORA-06512: at "SYS.UTL_RECOMP", line 940
ORA-06512: at line 1

According to the unwrapped utl_recomp source, those lines refer to the following functions:

recomp_serial() 
-> recomp_parallel()                 line 940
-> select_invalid_parallel_objs()    line 830
-> dbms_stats.gather_table_stats()   line 260

Where, at line 260, there is:

  260      DBMS_STATS.GATHER_TABLE_STATS('SYS', 'UTL_RECOMP_COMPILED',
  261         ESTIMATE_PERCENT => DBMS_STATS.AUTO_SAMPLE_SIZE,
  262         NO_INVALIDATE => FALSE);

So, utl_recomp.recomp_serial truly invokes dbms_stats.gather_table_stats with the intention of gathering statistics for a table called SYS.UTL_RECOMP_COMPILED:

SQL> desc SYS.UTL_RECOMP_COMPILED;
 Name              Null?    Type
 ----------------- -------- ------------
 OBJ#              NOT NULL NUMBER
 BATCH#                     NUMBER
 COMPILED_AT                TIMESTAMP(6)
 COMPLETED_AT               TIMESTAMP(6)
 COMPILED_BY                VARCHAR2(64)

which contains the list of compiled objects; it tracks which objects were compiled and in which batch.

Anyway, the dbms_stats.gather_table_stats must honor its global preferences, which can be set to use concurrency.

References

This blog post was tested on 19.29.0-EE.

Links to the official Oracle documentation:

• UTL_RECOMP

• DBMS_STATS

Here's the list of the main “tricks” we used to make it work.

Allowed Logon Version

These are the settings from sqlnet.ora:

SQLNET.ALLOWED_LOGON_VERSION_CLIENT=11
SQLNET.ALLOWED_LOGON_VERSION_SERVER=11

The _CLIENT one sets the minimum authentication protocol version that clients connecting to this database must use. And the _SERVER one sets the minimum authentication protocol that the database will use when acting as a client.

So, now the old clients are allowed to connect.

Just be also aware that allowing older protocols also means allowing weaker protocols.

USE_SID_AS_SERVICE

In most cases, you want clients to use service name rather than SID when connecting to a specific PDB. This is because if you connect using SID= to $ORACLE_SID, then you’ll be connected to CDB$ROOT and not to the PDB (which is usually the intention).

But in this case, the client was an embedded device with a hardcoded connect string - so changing the tnsnames.ora or the connect string itself to use service name instead of SID was not an option.

We can convince the listener to consider SIDs as service names by setting the following parameter in listener.ora:

USE_SID_AS_SERVICE_LISTENER = ON

Static Listener Service

However, there is a complication when the database is also using db_domain. This domain becomes the suffix to every service name that our database is using.

One way around this is to create a static listener.ora entry like this:

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (GLOBAL_DBNAME = sfx)
      (ORACLE_HOME = /oracle/db_se/19.28.0/dbhome_1)
      (SID_NAME = sfxc)
      (SERVICE_NAME = sfx.example.com)
    )
  )

This will create a new service in the listener called sfx. It is also the SID used by our embedded client.

(SID_NAME = sfxc) is the actual $ORACLE_SID that is running from $ORACLE_HOME. And since we want to connect to a specific PDB, we can also specify the default service name for this particular PDB, which in this example is (SERVICE_NAME = sfx.example.com).

References

• ALLOW_LOGON_VERSION_CLIENT

• USE_SID_AS_SERVICE_listener

• SID_LIST_listener

ORA-38886: WARNING: Flashback database was disabled due to error when writing flashback database logs.

Apparently, this is not a new behavior, as there's a blog post about it from the year 2010 by John Hallas.

But there have been improvements since he wrote this post. In 19c, the flashback logging (alter database flashback on/off) can be executed while the database is in READ WRITE state (no need to bounce it anymore).

I also agree with John that a parameter, which would cause flashback to behave similarly to a MANDATORY archivelog destination, would be a welcome addition. To my knowledge, such a parameter does not exist.

Official Docs

Here's what the official docs say about that:

The following rules govern creating, retaining, overwriting and deleting of flashback logs in the fast recovery area:

• If the fast recovery area has enough space, then a flashback log is created whenever necessary to satisfy the flashback retention target.

• If a flashback log is old enough that it is no longer needed to satisfy the flashback retention target, then the flashback log may be reused or deleted.

• If the database must create a flashback log and the fast recovery area is full or there is no disk space, then the oldest flashback log is reused instead.

Test Cases

I decided to create a test case to replicate the issue of flashback disabling itself. According to the docs, it shouldn’t easily happen due to space pressure, except, it seems, if there are actual I/O errors due to out-of-space conditions.

In both test cases, there were no Restore Points in use (empty v$restore_point).

Case 1: mountpoint (2G) smaller than db_recovery_file_dest_size (10G)

NAME                           VALUE              COMMENT
------------------------------ --------------- ----------
db_recovery_file_dest          /fra                  2 GB
db_recovery_file_dest_size     10737418240          10 GB
db_flashback_retention_target  1440                 24 h

FLASHBACK_ON       LOG_MODE
------------------ ------------
YES                ARCHIVELOG

On the first try, when the space ran out, I got this in the alert log:

2025-09-25T23:01:41.098262+02:00                                       
Errors in file /oracle/diag/rdbms/orcl/orcl/trace/orcl_gen0_1207793.trc:                                                                      
ORA-38701: Flashback database log 10 seq 1 thread 1: "/fra/ORCL/flashback/o1_mf_nfccfnv4_.flb"                                                                                                                                                                                              
ORA-27072: File I/O error                                              
Additional information: 4                                              
Additional information: 1153                                           
Additional information: 90112                                          
2025-09-25T23:01:41.098454+02:00                                       
Deleted Oracle managed file /fra/ORCL/flashback/o1_mf_nfccfnv4_.flb

But it did not stop the database from continuing to work. The only side effect was, as documented, V$FLASHBACK_DATABASE_LOG.OLDEST_FLASHBACK_TIME shifted forward as the oldest flb log(s) got deleted (the docs say “reuse,” but according to this test case, old ones are deleted so that new ones can appear).

Note that df reported a little bit of space still left (~9mb):

[root@db-server fra]# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdg        2.0G  1.8G  9.1M 100% /fra

So, I decided to use dd to ensure there is absolutely no space left (by creating dummy file(s)):

[root@db-server fra]# df -h /fra
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdg        2.0G  1.9G     0 100% /fra

Finally I got:

Errors in file /oracle/diag/rdbms/orcl/orcl/trace/orcl_gen0_1207793.trc:
ORA-38701: Flashback database log 10 seq 1 thread 1: "/fra/ORCL/flashback/o1_mf_%u_.flb"
ORA-27044: unable to write the header block of file
Linux-x86_64 Error: 28: No space left on device

And a bit later, finally:

ORA-38886: WARNING: Flashback database was disabled due to error when writing flashback database logs.
ORA-38701: Flashback database log 10 seq 12 thread 1: "/fra/ORCL/flashback/o1_mf_%u_.flb"
ORA-27044: unable to write the header block of file
Linux-x86_64 Error: 28: No space left on device

And, to confirm that it was actually disabled:

SQL> select flashback_on from v$database;

FLASHBACK_ON
------------------
NO

Case 2: mountpoint (2G) > db_recovery_file_dest_size (1G)

Now, given the observations from the first case, I would imagine that this only happens on an OS-level error. The chance of an out-of-space OS error occurring is much smaller if db_recovery_file_dest_size is correctly configured—not like I did in the previous case just to prove a point.

NAME                           VALUE             GB_VALUE
------------------------------ --------------- ----------
db_recovery_file_dest          /fra
db_recovery_file_dest_size     1073741824               1
db_flashback_retention_target  1440

Now, we do get a warning like this, but nothing is disabled because of space pressure:

ORA-19815: WARNING: db_recovery_file_dest_size of 1073741824 bytes is 93.69% used, and has 67731456 remaining bytes available.
2025-09-25T23:48:47.961667+02:00
************************************************************************
You have following choices to free up space from recovery area:
1. Consider changing RMAN RETENTION POLICY. If you are using Data Guard,
   then consider changing RMAN ARCHIVELOG DELETION POLICY.
2. Back up files to tertiary device such as tape using RMAN
   BACKUP RECOVERY AREA command.
3. Add disk space and increase db_recovery_file_dest_size parameter to
   reflect the new space.
4. Delete unnecessary files using RMAN DELETE command. If an operating
   system command was used to delete files, then use RMAN CROSSCHECK and
   DELETE EXPIRED commands.
************************************************************************
Reclaimable space = 0 
Available space = 67731456 
Disk space limit = 1073741824 
See trace file /oracle/diag/rdbms/orcl/orcl/trace/orcl_m000_1211049.trc for usage information from V$RECOVERY_AREA_USAGE

I’ve let redo be generated for about 24h straight, and even after RVWR wrote about 50GB of data, the FRA (Fast Recovery Area) did not get disabled and no out-of-space errors occurred.

How to Avoid

Having sufficient space for your FRA and maybe adding an additional check to your monitoring system (which is what we just did) is probably the safest bet.

Also, as demonstrated, having a dedicated mount point (or ASM disk group) for db_recovery_file_dest and having a correctly configured db_recovery_file_dest_size can significantly reduce the chance of this happening.

For monitoring purposes, you might want to distinguish between databases where Flashback was previously enabled (and later disabled) versus databases where it was never enabled (nor intended to). Based on my limited testing on 19.28.0, I observed an interesting pattern: when Flashback Database is disabled, Oracle removes the oracle-managed files but leaves behind an empty ./flashback/ directory within the FRA. In contrast, databases that never had Flashback enabled don't have this directory at all.

Important caveat: This is my preliminary observation based on a quick test and should be validated more thoroughly before relying on it for monitoring.

The Problem

Users complained that database sessions were occasionally "frozen." Interestingly, these "frozen" sessions could not be killed. When DBAs used alter system kill session, those sessions stayed in the KILLED state.

Also, the load average of the machine where this was happening was suspiciously high all the time, even though no one complained about things running "too slow."

The Investigation

I'll keep this story brief: the sessions couldn't be killed because their dedicated server process was STOPPED. By STOPPED, I mean the Linux process was in a T state. You can replicate this by sending a SIGSTOP signal to any Linux process, like this:

kill -SIGSTOP 12345

So, why is Oracle stopping those sessions? Initially, I suspected it might be related to Oracle Resource Manager and cpu_count settings because the load was high. However, as far as I know, Oracle doesn't use STOP/CONTINUE signals for resmgr. I even ran a quick test in my lab using resmgr under heavy load and did not see any such signals being sent.

Observing the Signals

This is where the bpftrace comes into play. We can use it to detect when the signal is sent (or received) and who sent it. And it isn't really a voodoo, all you need is a simple bpftrace script:

#!/usr/bin/env -S bpftrace 

tracepoint:signal:signal_generate /args->sig == 19/ {
    time("%Y-%m-%d %H:%M:%S");
    printf(" SIGSTOP sent from %d (%s) to %d (%s)\n", pid, comm, args->pid, args->comm);
}

I was hoping to let this script run for a week or so and then see which processes were sending SIGSTOP signals. But to my surprise, there were tons of such signals, many of them within the same second.

When I tried to see details of the process that sent the signal (the pid is printed using the script above), I noticed that such a process no longer existed. Even when I added printing info about the pid into the bpftrace script, like this:

system("cat /proc/%d/stack", pid);

I noticed that most of the time the process did not exist anymore when my script reached this call. So, they were very short-lived processes. Using a similar approach (reading PPid from /proc/pid/status), I found the parent process that was spawning them. And lo and behold, the parent process was the Oracle database session process itself!

Turns out, the Oracle database process was spawning a child process every second or so to send a STOP signal, do something, then send a CONTINUE signal again.

Why would Oracle do that?!

The Why

To understand why, I tried profiling a database session, which was waiting on SQL*Net message from client for a while now but was still receiving STOP/CONTINUE signals.

I’ve blurred out functions that were named like Something::doSomething. Notice how it looks like C++ code? If I remember correctly, most of Oracle's core is written in C. Could this really be Oracle code?!

Turns out that it is not. Inspecting shared libraries used by this process, like this:

cat /proc/12345/maps

proved, that Oracle process was using .so libraries from /opt/some-other-product.

So, it's not even an Oracle code; it's code from a product, that hooked Oracle database sessions in order to extract session-related info in a consistent manner.

Schema Only Accounts

They were introduced in 18c, and they allow us to create a user (who is the owner of objects, such as tables or views) without a password. Such a user cannot log on to the database.

It's simple to create such an account:

SQL> create user MYAPP no authentication;

Or, if you already have an existing user, you can modify it:

SQL> alter user MYAPP no authentication;

On older databases, we could achieve a similar result by setting an “impossible” password hash, as explained on Pete Finnigan's Oracle Security Weblog.

Note that traditionally, on older databases, simply revoking CREATE SESSION from the MYAPP user and then locking the account doesn’t really work because then you cannot log in to this schema via a proxy user, as explained in the next chapter.

So, now we have a user who has access to all the objects it owns - but can't log on.

Oracle Proxy Users

These are the answer to how developers can create/modify the objects owned by MYAPP user. Every developer should have his/her own account on the database. If there are many developers/users working on many databases, then maybe consider using externally authenticated users (such as those from MS Active Directory/Kerberos, Radius, etc). The point here is that, regardless of how developers' accounts are authenticated, there should be a database user for each developer.

Let's suppose that we have a developer named SCOTT. So we create a traditional database account for him:

SQL> create user SCOTT identified by "tiger";

Notice how we didn't grant anything to user SCOTT? At this point, he cannot even connect to the database, because he's lacking the CREATE SESSION privilege. Which we don't need to grant. Now, for the fun part:

SQL> alter user MYAPP grant connect through SCOTT;
SQL> grant CREATE SESSION to MYAPP;

We only allowed the schema owner, MYAPP, to connect through SCOTT. So, if SCOTT wants to use sqlplus to work on the MYAPP schema, he would connect like this:

$ sqlplus scott[myapp]/tiger@MYDB
SQL> select user from dual;

USER
-----------
MYAPP

So, he used his personal password to connect to the database, and yet he is working as the MYAPP user with all the permissions that the MYAPP user has.

Database Design and "API Schemas"

We already have MYAPP, which is the main schema owner with all the data, that is, all the tables. It can also contain packages and views with logic that is common to all API schemas. However, only developers, using proxy users, can connect. What about application servers and other end-user applications that need data from this schema?

Most importantly, no end-user should connect to MYAPP directly. Instead, create API schemas, such as, say MYAPP_GUI. There could be others, e.g., MYAPP_CLI. This API schema should have views and packages that are essentially wrappers for what MYAPP provides. This allows access to only the parts of MYAPP that are relevant for the interface and, equally important, exposes the interface so that its users can easily access it. For example, some libraries in certain languages have difficulty specifying RECORDs as parameters to package functions.

If clients (that is, applications) connect to those API schemas directly, we have already made a step forward from where this blog started. But we should still go a step further.

We can create those API schemas with no authentication in the same way we did with our main MYAPP schema. Then we can let end users connect using their own usernames and grant each of them roles, which allow them to execute actions on API schemas. After logging on, they would do something like:

SQL> alter session set current_schema=MYAPP_GUI;

Application Servers & Connection Pools

Traditional applications, such as Oracle Forms, tended to allow login using an end-user database account. And everything described above "just works." However, today, most web-based applications prefer to have a connection pool.

The problem with a connection pool is that it establishes a fixed number of connections, e.g., 30 connections. All of them are established with the same database user. So far, this is how we’ve tackled this problem:

About ten years ago, I wrote a connection pool compatible with the WildFly application server, which establishes connections with a user that has no privileges at all. Let's call it POOL_USER. The only privilege that this POOL_USER has is CREATE SESSION. We let it establish all the connections with this user. However, whenever we take a connection from the pool, we call:

oracle.jdbc.OracleConnection.openProxySession(), which essentially does the same as we showed with the sqlplus example earlier in the post. And whenever we return a connection to the pool, we call conn.close(OracleConnection.PROXY_SESSION), which returns to the unauthenticated context of being logged in as our POOL_USER, no longer as the real end-user. For more info, check Java Doc.

While this worked, it had a drawback - most Java applications would release database connections as quickly as possible (according to Java best practices). But that means developers can't use PL/SQL session state because it is lost after every page load. So, a few years later, I wrote H2DPool, which matches an HTTP session with one or many database sessions. As long as the user is logged in, the user has a database session established. That way, we're logging in as end-users and utilizing all the power of PL/SQL. And yes, this is no longer really a "pool," even though I named it so, because connections are established on application logon and closed on logout (or on session timeout).

Final Thoughts

This year, my colleague took my work on H2DPool even further and added support for OIDC, which may be the topic for another blog post. We never made those pool implementations publicly available; I just wanted to illustrate one of the possible ways to stay true to the guidelines presented in this blog in today’s world where 2FA, central user management and similar requirements are expected from most applications.

You can also read about topics from this post on oracle-base.com. Maybe the principles discussed here were considered utopian there back in 2005, but by sharing those ideas, evidently, they can eventually become reality.

• First, some are afraid that certificates must be involved and consequently maintaining a PKI infrastructure is deemed too complicated.
• Second, some are afraid of licensing issues, especially in respect to Standard Edition

Let me show you how none of those are true (certs certainly can be involved, but they're not necessary for simple setups).

License Requirements

To quote Database Licensing Information User Manual, Release 23:

Note: Network encryption (native network encryption, network data integrity, and SSL/TLS) and strong authentication services (Kerberos, PKI, and RADIUS) are no longer part of Oracle Advanced Security and are available in all licensed editions of all supported releases of Oracle Database.

I've also checked the same document for versions 19c and 12.1 and they both say the same thing.

Quick disclaimer: this blog simply reflects my understanding of Oracle's licensing documentation, so keep in mind that it is Oracle License Management Services (LMS) that always has the final authority regarding licensing matters.

That being said, it is also certainly a good idea to double check licensing restrictions when you hear about encryption or compression in regard to Standard Edition.

Server Configuration

So, let's setup Native Network Encryption as this is the simplest way of enabling encryption over SQL*Net. This method does not require any certificates and, in most cases, no client configuration. To enable it, we simply need to modify our sqlnet.ora file on the server. Like this:

SQLNET.ENCRYPTION_SERVER = REQUESTED
SQLNET.CRYPTO_CHECKSUM_SERVER = REQUESTED

Or maybe like this if we want to be really strict about it:

SQLNET.ENCRYPTION_SERVER = REQUIRED             # default=ACCEPTED
SQLNET.ENCRYPTION_TYPES_SERVER = (AES256)       # default=all available algorithms
SQLNET.CRYPTO_CHECKSUM_SERVER = REQUIRED        # default=ACCEPTED
SQLNET.CRYPTO_CHECKSUM_TYPES_SERVER = (SHA512)  # default=all available algorithms
SQLNET.ALLOW_WEAK_CRYPTO_CLIENTS = FALSE        # default=TRUE

Defaults for SQLNET.ENCRYPTION_SERVER and SQLNET.CRYPTO_CHECKSUM_SERVER is ACCEPTED. Which would mean that if a client REQUESTs or REQUIREs encryption then the server will be happy to encrypt the traffic. In our example though, as recommended for maximum security in official documentation, we only allow to establish encrypted connections, thus we specified REQUIRED. This is also why client configuration is not required in most cases: client also has default value for those parameters as ACCEPTED.

SQLNET.ALLOW_WEAK_CRYPTO_CLIENTS is more of a backward compatibility setting. If you need to support older clients then you may probably need to keep it on default value (TRUE).

Regarding algorithms, there is not much choice really. On 19c, there is AES for encryption and SHA for checksum. You can only require specific key lengths (e.g AES256) and hash sizes (e.g. SHA512). Well, to be exact, there are other algorithms (RC4, 3DES) supported, but they're also deprecated.

That's really, in essence, all there is to it.

Client Configuration

Same settings can be used on client, just replace _SERVER with _CLIENT. So, if you're only in control of the client, you can still have connection encrypted, even if your DBA did not touch the sqlnet.ora file. Simply set SQLNET.ENCRYPTION_CLIENT and SQLNET.CRYPTO_CHECKSUM_CLIENT to either REQUESTED or REQUIRED.

Is it Working?

Hackers will of course want to use tcpdump/Wireshark or similar to make it sure, but most of those people probably aren't reading this blog post as they already have some kind of encryption in place. For DBAs who trust performance views, you can simply do a following query:

SELECT network_service_banner
    FROM v$session_connect_info
    WHERE sid=sys_context('userenv','sid');

If enabled, you'll see (among other rows) those two:

NETWORK_SERVICE_BANNER
------------------------------------------------------------------------------------------
...
AES256 Encryption service adapter for Linux: Version 19.0.1.0.0 - Production
SHA1 Crypto-checksumming service adapter for Linux: Version 19.0.1.0.0 - Production
...

What About Certificates?

... What you seek is called Transport Layer Security (TLS) and it can be configured so that either only server has a certificate or both (client+server). You can also authenticate clients (and servers) this way.

As I prefer to keep my blog posts concise and on-topic, I'll only provide a link to the documentation instead of discussing TLS in this post.

References

• Configuring Oracle Database Native Network Encryption and Data Integrity
• Transport Layer Security (TLS)
• Database Licensing Information User Manual, Release 23

Those alternatives can sometimes yield a better execution plan and lateral join can sometimes yield a more readable code. So, as with anything - it depends. Let's consider a few examples.

Example

Suppose we'd like to get a list of all departments (DEPT) and for each one we'd also like to display salary and commission of an employee (EMP), who was the first hire in a given department:

select /*+ GATHER_PLAN_STATISTICS */ d.*, x.sal, x.comm
   from dept d
   join lateral (
      select sal, comm
         from emp e 
         where e.deptno = d.deptno
         order by e.hiredate asc
         fetch first 1 row only
   ) x on 1=1;

    DEPTNO DNAME          LOC                SAL    COMM
---------- -------------- ------------- -------- -------
        10 ACCOUNTING     NEW YORK       2450.00
        20 RESEARCH       DALLAS          800.00
        30 SALES          CHICAGO        1600.00  300.00

Explain Plan

select * from table(
    dbms_xplan.display_cursor('20p9d49b4q2j7', 0, 'ALLSTATS'));

-----------------------------------------------------------------------
| Id  | Operation                 | Name            | Starts | A-Rows |
-----------------------------------------------------------------------
|   0 | SELECT STATEMENT          |                 |      1 |      3 |
|   1 |  NESTED LOOPS             |                 |      1 |      3 |
|   2 |   TABLE ACCESS FULL       | DEPT            |      1 |      4 |
|   3 |   VIEW                    | VW_LAT_2D0B8FC8 |      4 |      3 |
|*  4 |    COUNT STOPKEY          |                 |      4 |      3 |
|   5 |     VIEW                  |                 |      4 |      3 |
|*  6 |      SORT ORDER BY STOPKEY|                 |      4 |      3 |
|*  7 |       TABLE ACCESS FULL   | EMP             |      4 |     14 |
-----------------------------------------------------------------------

dbms_xplan.display_cursor actually returns more columns - I'm only displaying Starts and A-Rows here to make a point. Those two rows are only available in execution plan output if you use /*+ GATHER_PLAN_STATISTICS */ (as I did in this example) or set statistics_level=ALL.

Letter "A" in A-Rows stands for Actual - it is the actual number of rows returned by the operation.

Starts tells us how many times the subquery started executing. So, according to line with Id=2, full table scan on DEPT ran once and returned 4 rows. In line Id=3 we see that our subquery was executed 4 times. All those executions together returned 3 rows (one of the departments has no employees, which is why line with Id=1 says only 3 rows were returned after JOIN).

Alternatives

Let's consider a few alternative solutions to our JOIN LATERAL example, just to give you ideas on what you can try if you find yourself hunting for a different execution plan. None of those is best; you can only get the best one if you study your specific execution plans and maybe compare the amount of buffer gets required.

Scalar Subqueries

The query that sparked the initial debate which eventually lead to this blog post was something like this:

select d.*,
      (select x.sal  from emp x where x.deptno = d.deptno order by x.hiredate asc fetch first 1 row only) as sal,
      (select x.comm from emp x where x.deptno = d.deptno order by x.hiredate asc fetch first 1 row only) as comm,
      /* ... */
   from dept d;

It can be "bad" (performance-wise) to execute subquery for each row, but it is twice as "bad" to do it twice for each row. Notice that JOIN LATERAL version accesses EMP table half as many times. Also, using expressions such as this example can only return 1 row whereas join lateral subquery can return any number of rows.

Eagle-eyed observers will probably also notice that our initial example would require left join lateral to match results of this alternative.

Analytic Functions

Using analytic functions we can read both tables only once, like this:

WITH emp_w AS (
   SELECT deptno, sal, comm,
          ROW_NUMBER() OVER (PARTITION BY deptno ORDER BY hiredate ASC) AS rn
    FROM emp
)
SELECT d.*, e.sal, e.comm
FROM dept d
LEFT JOIN emp_w e ON d.deptno = e.deptno AND e.rn = 1;

Correlated Subquery

This one may be the simplest one to understand:

SELECT d.*, e.sal, e.comm
FROM dept d
JOIN emp e ON d.deptno = e.deptno
AND e.hiredate = (SELECT MIN(hiredate)
                  FROM emp
                  WHERE deptno = d.deptno);

Final Thoughts

JOIN LATERAL is a powerful feature supported by many relational databases as it is also part of ANSI standard. On Oracle, it is supported since version 12c. However, I do believe that it is important to be aware of basic performance aspects of its usage.

How can we keep DDL changes of production database in git/mercurial repositoy? Application schemas of production database can be migrated using tools such as FlyWay or Liquibase which generally keep their history under version control.

But what about ad-hoc changes that happen on production? Wouldn't it be nice to have those, too, automatically under version control?

This blog post will provide a recipe to achive that using ddlfs. I decided to write examples in this blog post in Windows syntax, just because I'm still considering Windows support experimental (until I get more feedback), and I think writing this kind of blog post is a nice way to maybe catch some bugs before I mark the release as production-ready.

What is DDLFS?

It's an open-source filesystem which exposes every database object (such as packages, views, procedures) as an .sql file. Here is a screenshot on Windows of how it looks like in practice. Please visit ddlfs GitHub page for more detailed description.

Also, here is a video from which the screenshot was taken.

What about git/mercurial

They're both distributed version control systems, which, you, if you're reading blogs such as this one, should definitively be already familiar with. If not, here are the links to their manuals:

• Git Book
• Mercurial Guide

We do need to address one important detail regarding those two; They keep all their data under .git and .hg folder. So, if you have a project in, say, C:\Users\John\repos\my-project, there will be a .git or .hg folder in there.

Creating a project

Let's start by creating a project folder, say, F:\database\prod which contains .\ddlfs\ folder. Like this:

F:\>mkdir F:\database\prod\ddlfs
F:\>mkdir F:\temp

The F:\database\prod will contain .hg or .git folder and .\ddlfs\ folder will reflect the current state of database objects in the database.

The F:\temp is optional and can be deleted at any time when ddlfs is not mounted. We'll use it as temppath= (see details in the following section).

Mounting Database as a Filesystem

ddlfs -o ro,dbro,keepcache,temppathusername=SCOTT,password=tiger,database=dbhost:1521/service.name D:\database\prod\ddlfs

Regarding the folder D:\database\prod\ddlfs - this is mountpoint, it does not need to be a letter, e.g. X:\. Well, to be exact with terminology, on Windows, this "folder" becomes a junction after we successfully execute this ddlfs command:

F:\database\prod>dir
 Volume in drive F is Data
 Volume Serial Number is E277-8A48

 Directory of F:\database\prod

17/11/2024  14:02    <DIR>          .
15/11/2024  17:00    <DIR>          ..
17/11/2024  14:02    <JUNCTION>     ddlfs [\??\Volume{d6cc17c5-176c-4085-bce7-964f1e9f5de9}\]
               0 File(s)              0 bytes
               3 Dir(s)  21.358.292.992 bytes free

The only mandatory options are username, password and database. And everything described in this blog post should work if you'd only use those three. All the other options are added merely for performance optimization. On Linux, all mount options are described if you say man ddlfs, and all the mount options are also documented on the ddlfs GitHub page.

Let's discuss them in the context of using ddlfs for hg/git:

• ro: Do not allow changes to packages. Without this option, it is possible to open the .SQL files in .\ddlfs folder and save them. We intend to only read the current state of the database and, with this option, prohibit incidental changes.

• dbro: Let ddlfs assume your database is opened as READ ONLY. We need this assumption for better caching - once we read the contents of a package, we can trust that the state of this package didn't change even if we read it again after a while (until umount).

• keepcache: ddlfs keeps temporary files in which cached contents of database objects are kept. Those files can be reused between mounts, because cache validation is faster than reading all the objects again.

• temppath: Simply a path where previously mentioned cached contents are stored. Note that if you point it to a filesystem which relies on inodes - this folder can potentially have a lot of files, thus, very small ext4 may not cut it. Default is C:\Users\%USER%\AppData\Local\Temp\ on Windows and /tmp on Linux.

• filesize=-1: Probably the most important one; Using this option, ddlfs will always provide correct, actual, size of each file. Default is 0 - if this would be used, then files would have reported size=0 bytes until they're read. Once read, they report the correct size. Problem with correct size is that we need to export every database object in order to obtain every size. Which takes time. But we need to do that anyway in case of git/hg. It's also why we keepcache between the mounts.

You can optionally also specify schemas=MY_APP% to only include database objects belonging to schemas with name starting with MY_APP. You can also include list of schemas, like schemas=SCHEMA_1:SCHEMA_2:SCHEMA_3.

If you're scripting this, note that on Windows, ddlfs will go to background before everything is mounted, so before proceeding, make sure file D:\database\prod\ddlfs\ddlfs.log exists.

Git

git init is needed only the first time to create .git repository.

F:\database\prod>git init
Initialized empty Git repository in F:/database/prod/.git/

F:\database\prod>git config core.autocrlf false

F:\database\prod>git add ddlfs

core.autocrlf=false is here because we don't want git to care about line endings - we want it to add files exactly as they are, without line feed conversions (note that we've mounted .\ddlfs as read only).

The add part will take quite some time if your schemas are huge (it needs to export every object). Especially if you chose to log in using SYS user, which has access to all the objects in the database.

It will take the first time the longest (to build up a cache in temppath). Once this is built, we'll only deal with changes, which is way faster. This is also one of the differences between using custom export scripts (such as datapump metadata export, dbms_metadata.get_ddl or any other similar method - those methods will read everything they're exporting each time they're exporting it. Using ddlfs keepcache we're reusing everything that hasn't changed yet (according to all_objects.last_ddl_time).

Waiting on command to complete without knowing what it is doing and how much progress it's already made usually leads to lacking trust in any given software. So here's how you can monitor what is going on if you'd like:

You can run ddlfs in foreground by specifying -f flag to ddlfs command on Linux or use ddlfs-fg command (instead of ddlfs) on Windows. Furthermore, you can add loglevel=DEBUG to mount options. This way you'll see each object that is accessed by git printed to the console.

Once this is complete, simply commit (and maybe also push) and umount. On Windows, you need to be Administrator to do umount:

F:\database\prod>git commit -m "daily commit"

F:\database\prod>ddlfs-umount

Mercurial

Now, let's do the same with Mercurial, this time as a Linux example:

$ ddlfs -o... ./ddlfs
$ hg init
$ hg add ddlfs
$ hg commit -m "daily commit"
$ fusermount -u ./ddlfs

Replace three dots in the first command with mount options as described for git.

Also note, on Linux, you can mount/umount as any user, whereas on Windows, you need to be Administrator to umount.

Final Thoughts

Everything described so far is free and open-source. It is, however, also optionally integrated in my company's commercial offering Abakus Backup Server which uses physical standby databases to maintain physical backup copies of Oracle Databases using deduplication to save space.

Also keep in mind that ddlfs 3.x is not yet considered production ready because there has been many changes in order to accommodate both, Linux and Windows with the same source. So, feel free to report any issues on ddlfs GitHub page.

And if anyone happens to "convert" this article into an open-sourced generic script (e.g. bash or powershell), let me know, I'd be happy to publish a link to it.

The Query

The query which raised the question doesn't need much explaining. It's a simple example for demonstration purposes:

select /*+ ORDERED */ *
    from tab_first tf
    join tab_second ts on ts.id_first=tf.id_first;

---------------------------------------------------------------------------------
| Id  | Operation          | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |            |       |       |     4 (100)|          |
|*  1 |  HASH JOIN         |            |     6 |   978 |     4   (0)| 00:00:01 |
|   2 |   TABLE ACCESS FULL| TAB_FIRST  |     3 |   225 |     2   (0)| 00:00:01 |
|   3 |   TABLE ACCESS FULL| TAB_SECOND |     6 |   528 |     2   (0)| 00:00:01 |
---------------------------------------------------------------------------------

The Question

Which of the two tables is accessed first?

Is it tab_first because it is explicitly requested to be the leading table or is it tab_second because it is displayed as the last line in execution plan output?

Well, the correct answer is tab_first, but what prompted the question was actually the awareness of the general rule of the order in which execution plans are supposed to be read: from bottom up - starting with the deepest entries (leaf nodes). And, if we'd follow this rule to the letter, the tab_second would be accessed first.

This general rule comes from the way the execution plan is executed. Maybe it can help thinking of it this way: it all starts with root operation (id=0), but in order to complete this root operation it needs full or partial results from its children operations. And the same goes for all children that have children until we get to leaf nodes. That is why we're generally reading the plan as discussed earlier.

Let's first understand how Hash Join and Nested Loops operations work in order to better understand when is which table accessed.

Hash Join

This join method is generally used when joining two larger row sources (two tables in our example) using equality operator.

The optimizer will use statistics to determine which of the two row sources is smaller than the other and proclaim it to be the build table. We can also force specific table to be the build table by explicitly specifying the join order using hints (ORDERED or LEADING).

The other row source (generally the bigger one) is considered the probe table.

In our example, first_table is the build table and second_table is the probe table.

Hash Join will read the whole build table ("smaller one") first and calculate hash on joined columns for each row. The hashes are stored in PGA where each hash value has associated linked list of rows for which the same hash has been calculated.

The probe table (bigger one) is being read only after this hash -> linked list memory structure is built in PGA. This memory structure is probed by each record in the probe table.

So, in our example, tab_first is read first, and after that the tab_second is read.

Nested Loops

Let's now consider an execution plan with Nested Loops such as this one:

---------------------------------------------------------------------------------
| Id  | Operation          | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |            |     6 |   102 |     6   (0)| 00:00:01 |
|   1 |  NESTED LOOPS      |            |     6 |   102 |     6   (0)| 00:00:01 |
|   2 |   TABLE ACCESS FULL| TAB_FIRST  |     3 |    30 |     2   (0)| 00:00:01 |
|*  3 |   TABLE ACCESS FULL| TAB_SECOND |     2 |    26 |     1   (0)| 00:00:01 |
---------------------------------------------------------------------------------

Nested loops are probably the easiest to understand: for each row in tab_first loop over all rows in tab_second to find matching rows. tab_first would be considered outer table and tab_second would be considered the inner table in terms of Nested loops terminology.

In regards of when is which table accessed:

1. first, a batch of rows is read from tab_first
2. for each of the rows in this batch the matching rows in tab_second are found

So, tab_first is accessed first, then tab_second is iterated as many times as there is rows in first batch, then tab_first is accessed again for the next batch.

To put it differently, full table scan of tab_first is done once. But by the time it is done, the full table scan of tab_second was already done as many times as there is rows in tab_first.

Connecting the Dots

In the similar manner, if we'd have a multi-table join with nested loops, then all three tables would be accessed in described alternating manner and the execution plan would look like this:

----------------------------------------------------------------------------------
| Id  | Operation           | Name       | Rows  | Bytes | Cost (%CPU)| Time     |
----------------------------------------------------------------------------------
|   0 | SELECT STATEMENT    |            |     8 |  2008 |    12   (0)| 00:00:01 |
|   1 |  NESTED LOOPS       |            |     8 |  2008 |    12   (0)| 00:00:01 |
|   2 |   NESTED LOOPS      |            |     6 |   978 |     6   (0)| 00:00:01 |
|   3 |    TABLE ACCESS FULL| TAB_FIRST  |     3 |   225 |     2   (0)| 00:00:01 |
|*  4 |    TABLE ACCESS FULL| TAB_SECOND |     2 |   176 |     1   (0)| 00:00:01 |
|*  5 |   TABLE ACCESS FULL | TAB_THIRD  |     1 |    88 |     1   (0)| 00:00:01 |
----------------------------------------------------------------------------------

Hopefully, I managed to explain that reading the execution plan from bottom up, starting with the deepest entries (leaf nodes), is correct. But it does not mean that one operation must complete for the next to begin. And the operations displayed as siblings (tab_first and tab_second in our examples), don't necessarily execute in any specific order.

And if you need more food for thought on the subject - consider thinking about execution plans of parallel queries, pipelined functions and adaptive plans.

References

• Official doc: Oracle Documentation - Joins

We should use it every time we change $ORACLE_HOME. That means after migrations and after out-of-place upgrades, which is generally the preferred method of upgrading. You can read more on downsides of in-place patching on Mike Dietrich's blog.

This script is available in versions >= 19c.

OPatch

Interestingly, OPatch directories are fixed automatically on instance startup without running any scripts or commands other than startup. This applies only to OPATCH_%_DIR directories. So, it should be safe to run OPatch even before you run utlfixdirs.sql.

Regardless, I still believe it to be a good practice to fix directories before datapatch because .sql scripts that are run by datapatch could, in theory, need something from other, non-opatch directories (e.g. maybe from SDO_DIR_ADMIN directory).

READ ONLY PDBs

All Oracle maintained directories that point to paths in $ORACLE_HOME are created/replaced by Oracle using sharing=metadata. So, after running utlfixdirs.sql you will have those directories fixed even in PDBs that are open as READ ONLY. That includes PDB$SEED.

Example:

SQL> @?/rdbms/admin/utlfixdirs.sql

Container: CDB$ROOT

Current  ORACLE_HOME: /oracle/db_ee/19.24.0/dbhome_2
Original ORACLE_HOME: /oracle/db_ee/19.24.0/dbhome_1

DBMS_OPTIM_ADMINDIR
...OLD: /oracle/db_ee/19.24.0/dbhome_1/rdbms/admin
...NEW: /oracle/db_ee/19.24.0/dbhome_2/rdbms/admin
DBMS_OPTIM_LOGDIR
...OLD: /oracle/db_ee/19.24.0/dbhome_1/cfgtoollogs
...NEW: /oracle/db_ee/19.24.0/dbhome_2/cfgtoollogs
ORACLE_HOME
...OLD: /oracle/db_ee/19.24.0/dbhome_1
...NEW: /oracle/db_ee/19.24.0/dbhome_2
ORACLE_OCM_CONFIG_DIR
...OLD: /oracle/db_ee/19.24.0/dbhome_1/ccr/state
...NEW: /oracle/db_ee/19.24.0/dbhome_2/ccr/state
ORACLE_OCM_CONFIG_DIR2
...OLD: /oracle/db_ee/19.24.0/dbhome_1/ccr/state
...NEW: /oracle/db_ee/19.24.0/dbhome_2/ccr/state
SDO_DIR_ADMIN
...OLD: /oracle/db_ee/19.24.0/dbhome_1/md/admin
...NEW: /oracle/db_ee/19.24.0/dbhome_2/md/admin
XMLDIR
...OLD: /oracle/db_ee/19.24.0/dbhome_1/rdbms/xml
...NEW: /oracle/db_ee/19.24.0/dbhome_2/rdbms/xml
XSDDIR
...OLD: /oracle/db_ee/19.24.0/dbhome_1/rdbms/xml/schema
...NEW: /oracle/db_ee/19.24.0/dbhome_2/rdbms/xml/schema

PL/SQL procedure successfully completed.

References

• Official doc: Creating Additional Data Dictionary Structures

The contents of this blog post were tested on database version 19.24. Feel free to leave a comment if you find that described behavior changes with newer versions.

Before we begin -

What is an SQL Plan Baseline?

A baseline for a specific query (sql_text) stores one or more execution plans for this query. You can load execution plans into a baseline from the shared SQL area, AWR and other sources (see the documentation for package dbms_spm for more details). And you can persist those baselines in SMB (short for "SQL Plan Management Base"), that is, in your database dictionary or unpack ("export") them into a regular table. This table can be exported/moved using Data Pump etc later on.

The baselines in SMB can be enabled and/or fixed. Optimizer will first consider only enabled fixed plans. If no enabled fixed plans exists, then all enabled (but not fixed) plans are considered.

Regardless of how many plans for a statement are enabled, Oracle can still try to find a better plan than what is stored in the baseline. If it finds such a plan, then you will have the option of testing and accepting the new plan. One baseline can have many accepted plans - in such case, the optimizer will choose between all accepted plans.

Is this feature available on Standard Edition?

Since 18c, partially, yes. The main limitation is that SE only allows to store a single SQL plan baseline per SQL statement. This was announced on blogs.oracle.com.

Single plan baseline also means no opportunity for plan evolvement and other fancy features which would require more than a single plan per statement. So, this blog post will only mention what is relevant for SE from here on.

1. Use Case: Upgrades

Have you ever successfully upgraded the database without even a warning, just to be awoken the next morning by the customer stating that their database is running very slow or even seems like it is "hang"?

There are many possible reasons for such experiences and the main one probably being insufficient or no testing at all. Nonetheless, at times, the reason happens to be that the execution plan changed for worse due to newer version of CBO (Cost Based Optimizer).

Sure, you may succeed within a minute by using something like /*+ OPTIMIZER_FEATURES_ENABLE=19.1.0 */ and resume your morning coffee. If not, you likely need to figure out what the execution plan was on the previous version.

There are many options on how to obtain the old plan, even without baselines. Those options include ASH, AWR, our APPM and other third party tools. You may also restore previous version of database and run query there, hoping to produce a better plan.

Anyway, none of this options is as fast as simply doing this:

declare 
   l_count number := 0;
begin
   l_count := dbms_spm.alter_sql_plan_baseline(
      sql_handle => 'SQL_b85e793d85b59754',
      plan_name => 'SQL_PLAN_bhrmt7q2vb5undd079936',
      attribute_name => 'enabled',
      attribute_value => 'YES');
   dbms_output.put_line('enabled [' || l_count || '] baseline plans');
end;
/

So, see, with baselines, your coffee won't even get cold by the time you're done fixing the execution plan.

Regarding sql_handle and plan_name parameters... You can find those values in dba_sql_plan_baselines view by searching for your statement's sql_text:

select sql_handle, plan_name
   from dba_sql_plan_baselines
   where sql_text like 'select ... %';

sql_handle identifies your SQL. Maybe we can also call this a baseline. Each can have one or more execution plans, identified by plan_name. And each execution plan can be enabled (though in SE, only one plan_name per sql_hande is allowed). You can see the exact plan like this:

select * from table(dbms_xplan.DISPLAY_SQL_PLAN_BASELINE(
   sql_handle => 'SQL_b85e793d85b59754',
   plan_name => 'SQL_PLAN_bhrmt7q2vb5undd079936'));

2. Use Case: OLTP Applications

Let's imagine an OLTP application, which is deployed to many customers. So, schema and queries are the same at every customer, but each customer has their own data.

Suppose the vendor provides collected baselines. Here are the use cases:

When application is first deployed, customer doesn't yet have any data or they have incomplete data. And so, they don't have the correct statistics. Initial execution plans may be off because of that.

So, on EE we can have baselines which provide a starting point. If database comes up with a better plan than what is in a baseline, we can evaluate/accept that.

On SE, we can have a regular table, containing all baselines. If users experience unexpected delays because of suboptimal execution plan for specific query, we can simply load (and mark as fixed) the vendor supplied, field tested, execution plan.

A quick note on why the title says "OLTP" applications; those are online transaction processing apps, which usually have live users who expect nearly real time response. They get frustrated if they click something and it doesn't respond within the time they deem necessary. More importantly, such workloads tend to use somewhat simpler queries, which generally should execute always the same way. It's not about providing the absolutely best plan possible, it's simply about providing good enough plans that users can rely on.

In contrast to OLTP, say, data warehouses, where it's generally expected to run background jobs which move or transform vast amounts of data and run highly complex queries which produce complex execution plans. Baselines can still be beneficial in such scenarios - just be aware that setting a fixed baseline in SE for such a complex query might do no good when, for example, one of the many referenced tables drastically changes data.

If you are a software vendor, I'd suggest that you think about deploying baselines as part of your software installation.

Collecting Baselines

One, quite simple, way of collecting the baselines is spfile parameter OPTIMIZER_CAPTURE_SQL_PLAN_BASELINES=true. You can fine tune which plans to capture by using DBMS_SPM.CONFIGURE (e.g. to capture only queries with specific parsing schema). This will automatically add baselines for repeatable queries and will only happen at query parse time.

Another way of manually collecting baselines (which works on SE) is by using DBMS_SPM.LOAD_PLANS_FROM_CURSOR_CACHE, like this:

set serveroutput on;
declare
   l_plan_count number := 0;
begin
   for l_plan in (select distinct sql_id, plan_hash_value 
                     from v$sql 
                     where parsing_schema_name='MY_APP')
   loop
         l_plan_count := l_plan_count + dbms_spm.load_plans_from_cursor_cache (
            sql_id => l_plan.sql_id,
            plan_hash_value => l_plan.plan_hash_value,
            fixed => 'NO',
            enabled => 'NO');                     
   end loop;
   dbms_output.put_line('Loaded [' || l_plan_count || '] plans.');
end;
/

Staging Baselines (= Exporting)

Once you load baselines into SMB, they reside in the data dictionary and, if enabled, will be considered by the optimizer. You can export them from dictionary to a regular table by:

declare
   l_plan_count number := 0;
begin
   dbms_spm.create_stgtab_baseline (
      table_owner => 'DEMO_OWNER',
      table_name => 'DEMO_STGTAB');

   l_plan_count := l_plan_count + dbms_spm.pack_stgtab_baseline (
      table_owner => 'DEMO_OWNER',
      table_name => 'DEMO_STGTAB');
end;

You can also verify if baselines were exported by querying the created table:

select count(*) from demo_owner.demo_stgtab;

If you prefer, you can remove loaded plans from dictionary. You can remove only those that were not enabled and/or only those belonging to specific parsing schema, etc:

declare
   l_plan_count number := 0;
begin
   for l_baseline in (select sql_handle
                        from dba_sql_plan_baselines
                        where enabled='NO'
                           and parsing_schema_name='MY_APP')
   loop
      l_plan_count := l_plan_count + 
         dbms_spm.drop_sql_plan_baseline(sql_handle => l_baseline.sql_handle);
   end loop;
   dbms_output.put_line('Dropped [' || l_plan_count || '] baselines');
end;
/

Now, you can even move those baselines to another database by using Data Pump to export DEMO_STGTAB into, say, baselines.dmp. Then import this in another database and from there into SMB of another database.

Specifically for Standard Edition, I recommend keeping only the baselines that you actually need in SMB and removing all others, before staging and exporting all of them. Firstly, this will allow you to collect new plans and secondly, there is no restriction in SE on how many plans can be exported in a table - here's my example:

select sql_handle, obj_name, max(optimizer_cost) as cost
   from spm_test.spm_export_tab
   group by sql_handle, obj_name
   order by max(optimizer_cost) desc

SQL_HANDLE                     OBJ_NAME                             COST
------------------------------ ------------------------------ ----------
SQL_9832fdfb8497d4c2           SQL_PLAN_9hcrxzf29gp6202feb565      19089
SQL_9832fdfb8497d4c2           SQL_PLAN_9hcrxzf29gp62cea8bf8c        759

References

You can find out more about Managing SQL Plan Baselines in:

• official documentation

• dbms_spm pl/sql package for SPM management

• dba_sql_plan_baselines view which displays all available baselines in SMB

It would generally make sense to move spfile first and controlfile after that as this is the order rdbms instance will access them. However, we intentionally won’t start with spfile as it is only possible to move spfile in OMF fashion if rdbms instance has already established connectivity with asm instance (rdbms instance is registered as rdbms client in asm instance).

controlfile

RMAN> restore controlfile to '+SSD' from '/oradata/ssd/ORCL/controlfile/o1_mf_f1f8q9j1_.ctl';
SQL> alter system set control_files='+SSD/ORCL/CONTROLFILE/current.260.960811081' scope=spfile;

So, we start with controlfile. First obvious thing here is to specify '+DG' location without full target filename. Note that you can specify current contolfile in from clause (provided that your instance is mounted). If you specify backup location instead of current controlfile then your restored controlfile will be flagged as backup control file and thus you would need resetlogs to open the database (or recreate controlfile from source). You can use asmcmd to obtain the actual filename to which controlfile was restored to.

spfile

RMAN> backup as copy spfile format '/home/oracle/orcl.spfile';
RMAN> restore spfile to '+HDD' from '/home/oracle/orcl.spfile';

$ rm $ORACLE_HOME/dbs/spfileorcl.ora
$ echo "SPFILE='+HDD/ORCL/PARAMETERFILE/spfile.257.960812067'" > $ORACLE_HOME/dbs/initorcl.ora

This procedure seems to only work if ASM instance knows which database it is talking to. Specifically, if your rdbms instance is listed in v$asm_client (on asm instance). We achieved this by moving controlfile into ASM before attempting to move spfile. If this would not be the case, then your spfile would be created in +DG/DB_UNKNOWN/ folder.

While created asm alias may be correct, the actual path is determined by the asm client instance. So, when you put spfile or controlfile into asm, it does not matter what is stored in them (for example, spfile has db_unique_name written in itself). What does matter is db_unique_name of asm client which restored (copied) the spfile or controlfile to asm.