Oracle E-Business Suite 12.2 and Online Patching
In September 2013, Oracle announced the general release of the latest version of Oracle E-Business Suite, 12.2. The previous 12.1 version was released May 2009 so it has been a long 4 and half year wait!
As with any major product version there are a very large number of technology and functional changes included in 12.2, but here we will focus on one of the more exciting and significant – online patching.
Applying E-Business Suite patches pre-12.2
For all previous versions of eBusiness Suite, applying a patch always meant downtime and the amount of downtime that would be required was directly related to the time it would take to apply the patch.
For small, one-off patches, downtime could be as little as 15 minutes, but for Family Pack upgrades or Maintenance Packs this could last for several hours or even days.
For true 24/7 businesses where little or no downtime can be tolerated this would either result in a loss of revenue for the time period in which the system was unavailable or a general reluctance to apply any patch or upgrade which would incur a significant amount of downtime.
During the testing phase, the prime focus of the Oracle DBA would be to tweak the running order of patches or adjusting various runtime options to make the patches go through as speedily as possible and to make the best use of hardware resources, and project time would have to be allowed for this activity.
Finally, downtime can only normally be arranged outside of normal working hours at evenings and weekends which can incur overtime costs.
Online Patching in 12.2 with (virtually!) no downtime
A paradigm shift occurs with E-Business Suite patching in 12.2. Patches can be applied with the system up and running and as such critical business operations are not interrupted. E-Business Suite will remain available during patching operations (for example: tax year end patches can be applied during a payroll run or expense reports can be entered while a Payables patch is applied).
The only downtime that is required is the time it takes to stop and restart the application tier to propagate the changes and perform cutover – thus whether you are applying an online patch or a major upgrade, the amount of downtime will always be the same. This makes downtime predictable and measureable in minutes rather than hours or days.
Previously where the prime concern when applying a patch was to maximise server resources to allow the patch to complete as quickly as possible, now the prime concern is that the patch does not consume too many resources to impact on the performance of the running application. The new 12.2 online patching tools automatically adjust the number of parallel workers and workload to ensure that this is the case. Patches can therefore be expected to take longer to complete, but this can now be done during the online day with the application up and running.
How does it work?
At Database Level
A new feature was introduced in Oracle Database 11gR2 called Edition-Based Redefinition. This feature is available to any business that has Oracle Database 11gR2 or later (whether they have E-Business Suite or not) but it was commissioned specifically by the Oracle Applications Division to support the online patching feature of 12.2.
In a nutshell, it provides a mechanism that allows for multiple versions (or ‘editions’) of schema objects to exist concurrently in the database, a ‘run’ edition and a ‘patch’ edition.
When an online patch is applied the running application is not affected because the changes are made to a separate ‘edition’ of the database objects and the application will always choose to connect to the ‘run’ edition. The database will automatically manage the copies of these objects during patching. At cutover, the ‘patch’ edition objects are propagated to the ‘run’ edition.
Edition-Based Redefinition is supported for all metadata objects (such as packages, procedures, views, triggers, synonyms and triggers) but not for data objects (such as tables and indexes).
To manage these objects, new “Editioning Views” have been implemented in 12.2 which isolates the running application from changes to the data model, while cross edition triggers ensure that new transactions entered into the system are upgraded in place
In this example, the APPS.WF_ITEMS synonym points to the new WF_ITEMS# Editioning view rather than to the APPLSYS.WF_ITEMS table as it would have done in previous versions.
At Application Level
When 12.2 is installed or an existing system upgraded to 12.2, a second copy of the application filesystem is created. All patches are automatically applied against the secondary copy of the filesystem while the running application continues to point to the non-patched, primary copy.
Two complete filesystems are always present:
- The Run copy, used by the current application.
- The Patch copy, which is either actively being patched or waiting for the next patching cycle.
Synchronisation between the two filesystem copies is automatically handled by the patching tools.
During cutover, the application is shutdown and pointed to the patched copy which them becomes the primary, so the two file systems are rotated during each patch cycle.
The cutover process can happen at any point after the patches are applied, although certain restrictions mentioned below mean that, in practice, the window between applying an online patch and then performing cutover will probably never be too long.
New Patching Cycle
In addition, patches can be aborted at any point before Cutover – this can be one, some or all of the patches currently in the process of being applied.
Sounds too good to be true – what’s the catch?
There is no such thing as a free lunch of course and there are a number of implications and restrictions with online patching:
- Since dual copies of the application filesystem are maintained in 12.2, this means that the storage footprint of the E-Business Suite application tier will be significantly higher than it was in previous versions (35GB for 12.1, 64GB for 12.2 – an 85% increase). This increase overhead is proportional to the number of environments in use at any given site, so for example a 12.1 enterprise with 5 instances (prod, DR, preprod, dev, UAT) would need to budget and plan for an additional 145GB (5 x (64-35)) of disk space on the application tier as part of a 12.2 upgrade project.
- At database level, the SYSTEM and SEED tablespaces require double their space allocation to support the Edition Based Redefinition feature (25GB to 50GB for SYSTEM and 5GB to 10GB for SEED).
- Since 12.2 online patching requires the use of the database Edition Based Redefinition feature which is only available in 11gR2, E-Business Suite 12.2 is not certified with any prior database version below 11gR2.
Disaster Recovery considerations
- An additional scenario of what would happen when invoking DR when a patch is mid-flight (i.e. applied online but not yet cutover) needs to be allowed for in the DR solution.
- Restrictions when a patch is applied but not yet cutover.
- Performing a clone of an environment which has a patch mid-flight is strongly discouraged by Oracle.
- A small number of certain concurrent programs will not run.
Impact on custom and bespoke code
- All custom and third party code must now adhere to the new logical data model of using the APPS synonyms for objects rather than referencing objects directly, otherwise the incorrect ‘edition’ of data or code may be used. This means an audit and potential re-development of non-standard code will be required as part of any 12.2 upgrade project. Oracle have provided tools that can be used to check the database data dictionary and application filesystem for any object that violates the new coding standards.
Do we have to adopt the new online patching method?
Some non-24/7 enterprises that can tolerate downtime may at this point be wondering if there is an option continue to use the old style of patching to avoid the overhead and restrictions mentioned above in 12.2.
An Oracle FAQ answers this best:
Q: Once I upgrade to Release 12.2, can I still apply patches in the traditional way?
A: No. All patches for Release 12.2 will be online patches. The traditional, pre-12.2 method of applying patches will not work.
The benefits of online patching are:
- Applications stay online during patching.
- Easier to negotiate downtime windows which are now predictable and brief.
The potentially negative implications are:
- Increased storage overhead.
- Additional complexity for DR.
- Certain restrictions on operations which can be performed after patch has been applied but before cutover.
- Impact to custom code.