Thursday, August 19, 2010

R12 Upgrade Best Practice and Few Known Issues

Overview of upgrade to R12

Upgrading an application from 11i to R12 involves, upgrading the database side, upgrading the middleware techstack and upgrading the application side.
Supported upgrade path for application side upgrade is as given below.












Upgrade Steps in brief


Here are the 4 simple steps, briefly presented below for upgrade. These steps are at very high level of abstraction. Detailed each steps are further down.

1) Understand installed components, system sizing information, NLS considerations

2) Prepare for upgrade using Upgrade Manual Script(TUMS).

3) Upgrading to R12. This includes upgrading the database and applying the required patches through AutoPatch.

4) Post-Upgrade process. Complete the upgrade process by applying the latest RUP patches to keep the system most current.



Upgrade steps in detail



1) Understanding installed components


 Technology Stack Components

            Rapid Install automatically installs and configures the required technology stack components for both the database tier and the application tier.

The database tier technology stack for both a new installation and for a system upgrade is based on Oracle10g Release 2.

The technology stack installed on the application tier includes, among other components:

- Oracle 10g Application Server (AS) 10.1.2

- Oracle 10g Application Server (AS) 10.1.3

- Oracle Developer 10g (includes Oracle Forms)

- Java (J2SE) native plug-in 1.5.0_08

- Java Developer Kit (JDK) 5.0


Memory Requirements

       To calculate the memory requirements for an upgrade, consider the following:

- Number of concurrent users

- Infrastructure requirements for multi-tiered architecture

For example:

A test upgrade of the largest Oracle production system (oraprod) used the following:

- Database tier machine – 48 GB of memory

- Application tier machine – 12 GB of memory

A test upgrade of the Vision database and application tier machine used 6 GB of memory.


Database Size

             To estimate the increase in required disk space for upgrading, consider the products, the number of languages being installed, and changes in the data model.

For example:

In a test upgrade of the largest Oracle production system (oraprod), the database increased 10-20 percent. In a test upgrade, the Vision database increased 5 percent. For guidelines based on an upgrade of the Oracle production system (oraprod), see E-Business Suite Release 12 Upgrade Sizing and Best Practices (Doc ID: 399362.1).



Database Backup

*** Its highly recommended that you back up your entire system before you begin the upgrade. ***



Database Initialization Parameters

Initialization parameters required at each stage of the upgrade may vary depending on when you upgrade your database. Review the requirements for these parameters before you begin. Refer to metalink note ID 396009.1 for initialization parameters.

Change the following initialization parameters as specified below for upgrade process. Once the upgrade process completes, reset the parameters back.


db_file_multiblock_read_count – Remove this parameter. (this is not required).

_db_file_optimizer_read_count = 8 (default setting is 8. Keep default setting).

job_queue_processes (set the value of this parameters equal to number of CPUs).

parallel_max_servers (set the value of this parameters equal to twice the number of CPUs).

pga_aggregate_target (refer to metalink note ID 396009.1 for recommended value).

Make sure that the temporary tablespace you have is locally managed and not dictionary managed. You can check this information using below query.

select CONTENTS,EXTENT_MANAGEMENT,ALLOCATION_TYPE from dba_tablespaces where tablespace_name=’TEMP’;


CONTENTS     EXTENT_MANAGEMENT       ALLOCATION_TYPE

————               —————–                              —————

TEMPORARY         LOCAL                                   UNIFORM

Else if the extent management is not local, you can drop and recreate temp tablespace.

NLS Upgrade Considerations

    For NLS considerations, please refer to Applications upgrade docs


Character Sets


         You have to be careful while selecting the character set for APPL_TOP. Depending on whether your Applications system connects to the database during the upgrade process, you may be able to select a new character set for the Release 12 APPL_TOP on the Rapid Install wizard upgrade screens. However, if you do, the new set must be either identical to, or compatible with, the existing database character set. If you change the character set in the APPL_TOP to one that is not compatible with the current database character set, the upgraded system will be corrupted.


SQL> create TEMPORARY tablespace TEMP tempfile ’ts_p_temp1.dbf’ size 2048M EXTENT MANAGEMENT LOCAL UNIFORM SIZE 1M;



2) Prepare for upgrade using Upgrade Manual Script(TUMS) and upgrading database

          R12 upgrade process involve replacing 11i Tech stack (9iAS & 806) to Fusion Middleware (10g Application Server)

Basic upgrade process involves Rapid Install & Autopatch

Rapid Install involves installing new R12 tech stack as mentioned in first point

Auto patch process involves upgrading E-Business Suite database compatible to R12 (Data Model)

Final upgrade process is of updating data model using enhanced version of AutoPatch

Minimum version from which you can upgrade to R12 is 11.5.7 and higher

Minimum database version from which you can upgrade to R12 is 9i

As per Oracle R12 Upgrade Documentation, Apps 11i Instance is classified in Two Categories based on Apps & DB Version

Category 1 – 11.5.7, 11.5.8, 11.5.9 (CU1), 11.5.10 (CU1)

Category 2 – 11.5.9 (CU2), 11.5.10 (CU2) or 11.5.10.2


Why we cannot upgrade database to 10.2.0.2 before for category 1 ?

This is because there is no Interoperability patch for above release & 10.2.0.2 was supported for 11.5.9 (CU2) & 11.5.10(CU2) only.


What are advantages of Upgrading database to 10.2.0.2 before R12 upgrade ?

Downtime can be broken down to two small downtimes (one for Database upgrade another one for R12 upgrade) and can be achieved during weekends or long weekends (depending on your system & resources)



 
Best Practices - Pre Upgrade
 
  • Use TUMS to eliminate the tasks that are not relevant for your system.
  • Use Shared File System for Multi-Nodes
  • Use Distributed AD for Multi-Node.
  • Estimate Tablespaces size (Note # 399362.1)
  • Modify Database Parameters which are discribed above, that will help in performance of upgade scripts.
  •  Perform these tasks in advance to reduce downtime in Upgrade ( Convert to Multi Org, Convert to OATM, Upgrade Database to 10.2.0.3)
  • Gather Statistics before upgrade.
  • Record timing for each step during Test Upgrade.
  • Add PL/SQL no compile option in R12 upgrade driver to save time during upgrade
  • Choose proper Batchsize and Workers for Autopatch ( e.g. for a 24 CPU DB Server Batchsize=10000 and workers=30)
  • Make sure you reset the Database parameters which were modified earlier.
  • Merge All NLS patches and apply them as single merged patch.
  • Merging all Post 12.0.4 upgrade patches (including the 12.0.6 R12
    RUP6 Patch) in which, we applied the rule of thumb of merging both "non-AD" and "non-FND"
    patches together and thereafter, to individually apply AD, FND and online-Help patches. Merging
    the patches did save a lot of time, both in limiting the number of adpatch runs, and also in limiting
    the number of times the actual adpatch maintenance steps were executed (i.e. compiling Apps
    schema, compiling jtf files, etc.).
  • Isolate post upgrade concurrent programs to a separate manager queue ( Note # 399362.1)

Few Known Issues

• Purging Non-Required Data From the "GL_INTERFACE" Table Dramatically Reduced
Overall Patch Runtime

               In reviewing the patch runtime worker jobs, we've clearly noticed that script "jaiglintfsrcmig.sql" was taking roughly 4 hrs 55 mins to complete. After carefully reviewing the script's SQL path, we noticed that the script was actually doing several updates on the "GL_INTERFACE" table. And this was occurring on an implementation where table "GL_INTERFACE" is (was) not being purged after data loads. We then proceeded to ask the client's Development team to identify the non required data from the "GL_INTERFACE" table, which was then purged prior to engaging the Apps upgrade. Purging the non-required data from the "GL_INTERFACE" table, turned out to save approximately 4 1/2 hours of upgrade time.

• Apache Lock File Issues


             In setting a second and third Mid-Tier nodes, we've successfully completed the configuration of a shared APPL_TOP and shared Tech Stack. Redhat GFS was used for the shared filesystem. The only issue faced with the shared filesystem, was that Apache would fail to startup due to the "adapcctl.sh" startup script timing out. After carefully diagnosing the issue (using the "strace" command to see where the "httpd" processes were hanging), we identified that the issue resulted in Apache trying to lock files on the GFS mountpoints. This appeared to be a generic issue with most shared filesystem when locking files take place (which is in fact an Apache requirement). The issue was fixed by configuring Apache parameters "s_lock_pid_dir", "s_pids_dir" and "s_web_pid_file" (in the context xml file) to point to local mount point "/var/tmp" (instead of pointing to shared mountpoints). These changes were made to all context xml files
for all nodes in the cluster.

• R12 Login Page not Working After Applying the 12.0.4 Patch


           After applying the 12.0.4 patch, for some reason, the R12 login page stopped working properly, more specifically, the R12 login page would loop between "/OA_HTML/AppsLogin" and "/OA_HTML/RF.jsp?" and consequently, the login page would not come up at all. After thorough troubleshooting, we determined that the FND tables and WF tables were not completely synchronized, thus making the "RF.jsp" go back to the AppsLogin page as user "GUEST" which in fact, does not have necessary responsibilities against WF tables. The problem was then fixed by running the "WF_LOCAL_SYNCH.VALIDATEUSERROLES" procedure.

• Intermittent Issues


                At one point, we were faced with two types of intermittent issues. The first was with respect to a report customization where the "Payables Open Interface Import" program used to fail intermittently with error "REP-0069: Internal error, REP-50002: Server is shutting down". After researching this issue, we've identified patch 6116405 from Metalink Note 549910.1 "Error REP-0069 and REP-50002 while Generating Reports through Concurrent Managers", which in the end, fixed the issue.

The other intermittent issue dealt with Apache producing errors of the form "404 not found in OA Framework pages". After carefully researching and diagnosing the issue, we narrowed it down to OC4J not behaving correctly and further more, identified that the error would occur only when OC4J oacore jvms were increased from 1 to 3. We then tracked down Metalink Note 731115.1, specifying to apply patch 7311892 to the 10.1.3 Oracle Home and also to add set context file parameter "s_oacore_jvm_start_options" to "-Doracle.ons.numprocs=", and of course to also run Autoconfig to propgagate all changes. This fixed the Apache errors of the form "404 not found in OA Framework pages".


Let me know if I missed anything or wants me add anything.... Take Care all.

No comments:

Post a Comment