What happens during adop phase=cutover ?

The phase adop phase=cutover , has many tasks.  

• Validations: First on primary node and then on all slave nodes in parallel
• Shutdown of services: Parallel on all slave nodes followed by primary node
• DB Cutover: Only on primary node, rest all slave nodes wait
• FS Cutover: Parallel on all nodes
• Startup of services: First on primary node and then on all slave nodes in parallel

Note that all these subtasks are completed sequentially one after the other i.e. each subtask should be complete on all nodes before proceeding to the subsequent subtask.

Object Invalidation post Grant/Revoke on Objects

GRANT and REVOKE on objects for the user is very common requirement in any Database projects and is no special in Oracle as well.  Especially DBAs have provided GRANT to users  on various objects (based on requirements). This was working all the way without causing any issues to the application availability.

But in Oracle E-Business Suite R12.2.x, Whenever granting privileges on an object belongs to APPS Schema cause an invalidations in the current edition. Hence if you are GRANTing privileges in the RUN edition has the potention to cause an impact on the application availability. To avoid such invalidation, use the procedure AD_ZD.GRANT_PRIVS.  (For more information on AD_ZD.GRANT_PRIVS, see Oracle Doc ID 1577661.1)

But good news for those who still want to use native GRANT and REVOKE in R12.2.x application instead of the above API. Oracle has fixed this.

Follow the below action plan.

• set the parameter _disable_actualization_for_grant=TRUE (with scope=spfile)
• Bring down the database.
• Apply the one-off patch 26654363 to the database. (This patch is part of ETCC, and hence chances are that this patch might have been already applied). 
• Start the database.

Now you will be able to use Native GRANT and REVOKE in a way you have practised all these years.


References:

• Granting Privileges On An Object May Cause Invalidations (Doc ID 1987947.1)
• Developing and Deploying Customizations in Oracle E-Business Suite Release 12.2 (Doc ID 1577661.1)
• Objects Still Get Invalidated After Granting Privileges For Edition Object After Applying Patch 26654363 (Doc ID 2392091.1)

How to find StartCD used for installation of R12.2 ?

Run the RapidWizVersion command:

cd <startCD_location>/Disk1/rapidwiz

./RapidWizVersion > RapidWizVersion.txt   (RapidWizVersion.cmd on Windows)

RapidWizVersion.txt will have the StartCD version which is being used for R12.2 binary installation.

Incase if you are not aware of the StartCD location (due to some reasons), how can you find out StartCD version?

Answer: It is not possible to find out the StartCD version. But there is a workaround for this.

Unless otherwise upgraded by customer

If the customer is on FMW 11.1.1.7 you can assume that startCD50 was used.

If the customer is on FMW 11.1.1.6 you can assume that startCD49 or 48 was used.

If the customer is on FMW 11.1.1.9 you can assume that startCD51 was used.

At the moment, StartCD51 is the latest available for download for R12.2 installation.

How to find EBS CPU Patch Level in R12.1 or R12.2 ?

Most of the organizations have standard practise in applying the latest Patch Set Update (PSU) for the databases. But tend to ignore the security fixes for the mid-tier technologies due to lack of exposure in these technologies.

Below are the scripts to find out the CPU patches applied in the system. 

For 12.2 use the below script. 

set pagesize 100;

set linesize 141;

  

col "Patch" format A10;

col "Description" format A70;

select distinct bug_number Patch, decode(bug_number,

    '28421544','ORACLE APPLICATIONS RELEASE 12.2: CPU PATCH FOR OCT 2018',

    '28018169','ORACLE APPLICATIONS RELEASE 12.2: CPU PATCH FOR JUL 2018',

    '27468058','ORACLE APPLICATIONS RELEASE 12.2: CPU PATCH FOR APR 2018',

    '27040860','ORACLE APPLICATIONS RELEASE 12.2: CPU PATCH FOR JAN 2018',

    '26574498','ORACLE APPLICATIONS RELEASE 12.2: CPU PATCH FOR OCT 2017',

    '25982922','ORACLE APPLICATIONS RELEASE 12.2: CPU PATCH FOR JUL 2017',

    '25449173','ORACLE APPLICATIONS RELEASE 12.2: CPU PATCH FOR APR 2017',

    '25032335','ORACLE APPLICATIONS RELEASE 12.2: CPU PATCH FOR JAN 2017',

    '24390794','ORACLE APPLICATIONS RELEASE 12.2: CPU PATCH FOR OCT 2016',

    '23144508','ORACLE APPLICATIONS RELEASE 12.2: CPU PATCH FOR JUL 2016',

    '22614473','ORACLE APPLICATIONS RELEASE 12.2: CPU PATCH FOR APR 2016',

    '22133451','ORACLE APPLICATIONS RELEASE 12.2: CPU PATCH FOR JAN 2016',

    '21507429','ORACLE APPLICATIONS RELEASE 12.2: CPU PATCH FOR OCT 2015',

    '20953338','ORACLE APPLICATIONS RELEASE 12.2: CPU PATCH FOR JUL 2015',

    '20406630','ORACLE APPLICATIONS RELEASE 12.2: CPU PATCH FOR APR 2015',

    '19873050','ORACLE APPLICATIONS RELEASE 12.2: CPU PATCH FOR JAN 2015') Description,

     CREATION_DATE Applied

from ad_bugs

where bug_number in

('28421544','28018169','27468058','27040860','26574498','25982922','25449173','25032335','24390794','23144508','22614473','22133451','21507429','20953338','20406630','19873050') order by Description desc;

For 12.1 use the below script. 

set pagesize 100;

set linesize 141;

  

col "Patch" format A10;

col "Description" format A70;

select distinct bug_number Patch, decode(bug_number,

'28421543','ORACLE APPLICATIONS RELEASE 12.1: CPU PATCH FOR OCT 2018',

'28018146','ORACLE APPLICATIONS RELEASE 12.1: CPU PATCH FOR JUL 2018',

'27468057','ORACLE APPLICATIONS RELEASE 12.1: CPU PATCH FOR APR 2018',

'27040859','ORACLE APPLICATIONS RELEASE 12.1: CPU PATCH FOR JAN 2018',

'26574496','ORACLE APPLICATIONS RELEASE 12.1: CPU PATCH FOR OCT 2017',

'25982921','ORACLE APPLICATIONS RELEASE 12.1: CPU PATCH FOR JUL 2017',

'25449171','ORACLE APPLICATIONS RELEASE 12.1: CPU PATCH FOR APR 2017',

'25032333','ORACLE APPLICATIONS RELEASE 12.1: CPU PATCH FOR JAN 2017',

'24390793','ORACLE APPLICATIONS RELEASE 12.1: CPU PATCH FOR OCT 2016',

'23144507','ORACLE APPLICATIONS RELEASE 12.1: CPU PATCH FOR JUL 2016',

'22614470','ORACLE APPLICATIONS RELEASE 12.1: CPU PATCH FOR APR 2016',

'22133441','ORACLE APPLICATIONS RELEASE 12.1: CPU PATCH FOR JAN 2016',

'21507207','ORACLE APPLICATIONS RELEASE 12.1: CPU PATCH FOR OCT 2015',

'20953340','ORACLE APPLICATIONS RELEASE 12.1: CPU PATCH FOR JUL 2015',

'20406628','ORACLE APPLICATIONS RELEASE 12.1: CPU PATCH FOR APR 2015',

'19873049','ORACLE APPLICATIONS RELEASE 12.1: CPU PATCH FOR JAN 2015') Description,

CREATION_DATE Applied

from ad_bugs

where bug_number in

('28421543','28018146′,’27468057′,’27040859′,’26574496′,’25982921′,’25449171′,’25032333′,’24390793′,’23144507′,’22614470′,’22133441′,’21507207′,’20953340′,’20406628′,’19873049') 

order by Patch desc;

You can see the patches applied in the system. Go through each line and find out which is the latest among all rows. With this, you will be able to find out how much you lag interms of security update for your EBS application. 

As you understood,  this script is as of OCT 2018.  For future releases, you need to add patch numbers to this query. 

Upgrading Oracle E-Business Suite (EBS) to 12.2.8

R12.2.8 is the latest Release Update Pack(RUP) available for R12.2 version.  Insturctions for downloading and applying the patch can be found from the below MoS Note:

Oracle E-Business Suite 12.2.8 Readme (Doc ID 2393248.1)

Release 12.2.8 is cumulative. That means, as well as providing new updates,features,fixes it also includes earlier versions features as well.  As per the release notes, there are enhancements for stability, performance and security.

Let's us explore how to upgrade to this latest RUP version 12.2.8. Assumption is, you are already on one of earlier versions of 12.2.

Initiate an online patching Cycle.

adop phase=prepare 

There are pre-requisites you need to verify before the next steps.

FMW

• Check the versions of Fusion Middleware Technology Stack (FMW). If you are on 11.1.1.6 then definately you need to upgrade to either 11.1.1.7 (11gR1 PS6)  or 11.1.1.9 (11gR1 PS7) . 
• To upgrade to 11.1.1.7 (11gR1 PS6) use the Mos Doc 2002603.1
• To upgrade to 11.1.1.9 (11gR1 PS7) use the Mos Doc 1590356.1

Tech Stack Patches

Apply the latest application tier technology patches. Refer the Section 3 and 4 from the Doc ID 1594274.1 to identify and apply the patches to all the tech stack components. This Doc also helps provides instructions to verify the output of the latest EBS Techonology Codelevel Checker(ETCC) script. ETCC can be downloaded via Patch 17537119.  Make sure ETCC reports no patches missing. 

once all the pre-requisites met, apply the patch 26787767 (This is big patch. Download in prior to upgrade. Otherwise use wget.sh to download directly to your server).

Apply the 12.2.8 patch 

adop phase=apply patches=26787767

adop phase=finalize,cutover,cleanup

Post Update Steps

• Apply Oracle E-Business Suite Release 12.2.8 Online Help Patch 26787780 to the new Run file System

adop phase=apply patches=26787780 hotpatch=yes

• E-Business Suite is locked down - Verify Security configurations

After the RUP applied, E-Business Suite is locked down until the APPS DBA acknowledges the recommended security configurations. User with System administrator responsibility should login and then he will be redirected to the Secure Configuration Console. 

There are security guildesliness which you have to go through in detail as displayed in the console. Certain actions are autofixable and many requires manual actions from your side.

• Grant flexfield value set access to specific users (required)

This is a mandatory action items incase if you have not done in the previous 12.2.x releases. It's a new feature introduced in 12.2. It controls who can create or modify flexfield values in the Flexfield Values Setup window. For more details, check the 'Flexfield Value Set Security' chapter in Oracle® E-Business Suite Flexfields Guide, Release 12.2 (Part No. E22963). 

• Perform Product-Specific Steps (conditional)

There are product specific actions need to be performed post update. It is advised to loop correspending product teams from your organizations to verify and perform the changes. Also, these teams have to verify the documents to verify if oracle has released any patches for their corresponding module.

• Perform NLS-related Step (conditional)

Incase if you have additional launguates installed, you must apply the 12.2.8 NLS software patch for the other languages enabled. Refer the Doc Oracle E-Business Suite NLS Release Notes, Release 12.2 (Doc ID 1314621.1) for more details on NLS.

• Consider E-Business Suite recommended patches

There are recommended patches for E-Business Suite environment. Please have a visit to the Doc How to Find E-Business Suite Recommended Patches (Doc ID 1400757.1)

• Look and Feel

From 12.2.5 onwards the look and feel of the E-Business suite has been changed. It is not possible to keep the same look and feel from the previous to 12.2.5 versions.

References: 

1. Oracle E-Business Suite Release 12.2.8 Readme (Doc ID 2393248.1)
2. Document 2002603.1, Fusion Middleware Technology Stack of Oracle E-Business Suite Release 12.2 to 11.1.1.7 (11gR1 PS6).
3. Document 1590356.1, Fusion Middleware Technology Stack of Oracle E-Business Suite Release 12.2 to 11.1.1.9 (11gR1 PS7)
4. Oracle E-Business Suite Release 12.2: Consolidated List of Patches and Technology Bug Fixes (Doc ID 1594274.1)
5. Oracle E-Business Suite NLS Release Notes, Release 12.2 (Doc ID 1314621.1)
6. How to Find E-Business Suite Recommended Patches (Doc ID 1400757.1)

Is cmclean.sql safe to run in EBS R12.x?

There are bit and pieces for information on "cmclean.sql". Here, this is an attempt to place all information related to cmclean.sql in one place. 

There is a script, cmclean.sql, most Apps DBA has run atleast few dazon times in their life time to fix the concurrent manager start up issues either during maintenance activities or Migration/Upgrade activities. This script was initally develped to clean source data during EBS installation and not for general maintenance or any other activities.

Running CMCLEAN.SQL

1. Will not update/clean any pending requests.
2. It updates only the requests which are in either RUNNING OR TERMINATING status to COMPLETED with ERROR. But, here is the catch. It doesn't reschedule requests which have a periodic schedule attached to them.

So apps DBAs, who are aware of the issue, spend atleast 1 to 2 hours after every major maintenance activity on EBS, following up with Business people, to get these cmclean terminated request rescheduled.

Some DBAs, who are not aware of this limitation, end up with system which has some jobs missing and might cause the business team upset or spend good amount of time investigating whats wrong with the system. 

As per the MOS Doc ID 2183718.1; The script CMCLEAN.SQL is NOT recommended or supported with E-Business Suite (EBS) Release 12.

If you encounter such situations, either use cpadmin.sh (adadmin.sh style utility) or use the Concurrent Manager Recovery Wizard. 

Concurrent Processing Command-Line Utility

The command-line utility cpadmin consolidates various existing utilities for concurrent processing into a single menu-based utility. This adadmin-style utility can be used for multiple tasks, including:

• Manager Status: Use this option to display the statuses of all managers.

• Clean CP Tables: Use this option to clean up the concurrent processing tables. This utility replaces cmclean.sql.

• Set Manager Diagnostics: Turn diagnostics on or off for individual managers with this option.

• Manager Control: Use this option to send a request such as start, stop, or verify to an individual manager.

• Rebuild Manager Views: Use this option to rebuild the FND_CONCURRENT_WORKER_REQUEST and FND_CONCURRENT_CRM_REQUESTS views.

• Move Request Files: Change request log and output file locations with this option.

• Analyze Request: Use this option to analyze a concurrent request.

Complete the steps below to run the command-line utility and its maintenance tasks.

1. Set the environment.

You must set the environment in order to apply the environment variables that define your system. This task is common to many utilities. See Setting the Environment, Oracle E-Business Suite Maintenance Guide for the preparatory steps.

2. From any directory, start cpadmin with this command:

$ cpadmin.sh

The utility starts and prompts you for the APPS password (required).

3. Respond to prompts.

Supply the information requested by cpadmin prompts. Prompts unique to an option are described with the option.

When you complete the prompts, the Main Menu appears.

4. Choose one of the tasks listed above.

Enter 2 if you are running this utility in place of cmclean.sql

Read the above instructions and proceed accordingly. 

Now this the table has been updated for deactivated and terminated manager process. 

It also checks the runaway process for the Managers. 

Now, You have successfully fixed the concurrent manager issues. 

5. Exit the cpadmin utility.

For more information, please visit the Oracle Documentation here: http://docs.oracle.com/cd/E26401_01/doc.122/e22953/T174296T575591.htm#8159578

Concurrent Manager Recovery Wizard

NOTE: The Concurrent Managers have to be stopped before the Wizard is run. Also insure that listeners are not stopped before the managers are stopped.

Please run the Concurrent Manager Recovery wizard to address Concurrent Manager failures within the Oracle Applications Manager.

Using the Concurrent Manager Recovery wizard is the method to clear the errors upon bringing the Internal Manager back up.

For Concurrent Internal Manager failures, it is recommended to run the Concurrent Manager Recovery feature using the Oracle Applications Manager. This feature should be used for recovering from when the Internal Manager won't start. This is accessed from the Troubleshooting wizards available within Oracle Applications Manager logged in as the sysadmin userid. 

Navigate:

Oracle Applications Manager > Concurrent Managers OR Concurrent Requests > Site Map > Diagnostics and Repair > Concurrent Manager Recovery.

Reference:

Is the Script CMCLEAN.SQL Supported in EBS R12? (Doc ID 2183718.1)

Concurrent Processing - Command-Line Utility OR Recovery Wizard (Doc ID 134007.1)