RAC Attack - Oracle Cluster Database at Home/Patching Grid and Database Software

Prev: Create Database



Rolling Patches (e)
  1. Patching Grid and Database Software


Next: Clusterware Testing



Patching OverviewEdit

RA-patching-overview.png
  • Upgrades to the "base version" are very complicated and always use the full Oracle installer (runInstaller). Major new features are only introduced in new base versions.
  • Patch Sets are also installed with the full Oracle installer. Historically, each patchset was installed on top of the base version (top row in the illustration) by using runInstaller. However, starting with 11.2.0.2 the patch sets can be installed as a new installation without the base version. It is now recommended to perform Patch Set upgrades "out-of-place" in this manner. Sometimes new features are also included with Patch Sets (for example RAT data collection).
  • PSUs are installed with opatch. They include security updates and important bug fixes. They are released quarterly and always include the latest CPU.
  • CPUs are installed with opatch. CPUs include only security updates, and are also released quarterly. They cannot be applied after you have applied any PSU. (Until you upgrade to a new patch set or base version.)

Support Status and Known IssuesEdit

Before performing any installation or upgrade of Oracle, you should always check the Support Status and Known Issues for the release. Metalink note 161818.1 is always the starting point – open this note and review it. Next, follow the link for 11.2.0.X to metalink note 880782.1 and review that note. Finally, follow the link to note 880707.1 and review the known issues with Oracle 11.2.0.1 which is the version we will be using for this lab.

Practice Rolling ApplicationEdit

For this lab, the instructor has provided recent PSUs. PSUs and CPUs are collections of one-off patches. One-off patches can only be applied to an Oracle database in a rolling manner if they have been certified for rolling upgrades.


  1. Review the installation instructions. We're going to install three patches and you can find the README files at these locations:

    • /mnt/cdrom5/patch/psu6-db-12419378/12419378/README.html
    • /mnt/cdrom5/patch/psu2-gi-9655006/README.html
    • /mnt/cdrom5/patch/opatch-6880880/README.txt
  2. First we need to update the OPatch utility. Find patch 6880880 on the instructor-provided CDROM and unzip it directly into both the grid home and the database home. Before unzipping the file, backup the existing OPatch programs.

    $ oenv
    SIDs here are: grid +ASM1 RAC1
    ORACLE_SID = [RAC1] ? grid
    $ cd $ORACLE_HOME
    $ tar czvf ~/gi-opatch-backup_$(date +%Y-%m-%d).tgz OPatch/
    ...
    $ unzip -o /mnt/cdrom*/patch/opatch-6880880/p6880880_112000_LINUX.zip
    ...
    $ OPatch/opatch version
    Invoking OPatch 11.2.0.1.5
    OPatch Version: 11.2.0.1.5
    OPatch succeeded.
    
    $ oenv
    SIDs here are: grid +ASM1 RAC1
    ORACLE_SID = [grid] ? RAC1
    $ cd $ORACLE_HOME
    $ mv OPatch OPatch.backup
    $ unzip /mnt/cdrom*/patch/opatch-6880880/p6880880_112000_LINUX.zip
    ...
    $ OPatch/opatch version
    Invoking OPatch 11.2.0.1.5
    OPatch Version: 11.2.0.1.5
    OPatch succeeded.
    


  3. Repeat the previous step on node collabn2.

    $ ssh collabn2
    
    ... repeat previous step ...
    
    $ exit
    
  4. This new version of OPatch requires an "OCM response file" for certain operations. Use the OCM utility to generate this file. We don't want to configure OCM; leave your username blank and confirm that "YES" you don't want to enter any account information.

    $ $ORACLE_HOME/OPatch/ocm/bin/emocmrsp -output /home/oracle/ocm.rsp
    OCM Installation Response Generator 10.3.4.0.0 - Production
    Copyright (c) 2005, 2010, Oracle and/or its affiliates.  All rights reserved.
    
    Provide your email address to be informed of security issues, install and
    initiate Oracle Configuration Manager. Easier for you if you use your My
    Oracle Support Email address/User Name.
    Visit http://www.oracle.com/support/policies.html for details.
    Email address/User Name: <enter>
    
    You have not provided an email address for notification of security issues.
    Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
    The OCM configuration response file (/home/oracle/ocm.rsp) was successfully created.
    


  5. Find the Grid Infrastructure and Database PSUs on the instructor-provided DVD.

    $ cd /mnt/cdrom5/patch
    $ ls
    opatch-6880880  psu2-gi-9655006  psu6-db-12419378
    


  6. We know that these PSUs can be applied in a rolling manner, but verify this.

    $ for D in psu*/[0-9]*; do
    echo -n $D:
    cd $D
    $ORACLE_HOME/OPatch/opatch query -is_rolling_patch | grep rolling
    cd ../..
    done
    
    psu2-gi-9655006/9654983: Patch is a rolling patch: true
    psu2-gi-9655006/9655006: Patch is a rolling patch: true
    psu6-db-12419378/12419378: Patch is a rolling patch: true
    


  7. We also know that there won't be any patch conflicts (since these are the first patches we're installing) – but verify this too.

    $ oenv
    ORACLE_SID = [RAC1] ? grid
    [grid]$ cd psu2-gi-9655006
    psu2-gi-9655006[grid]$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./9655006
    
    ...
    Invoking prereq "checkconflictagainstohwithdetail"
    Prereq "checkConflictAgainstOHWithDetail" passed.
    
    [grid]$ oenv
    ORACLE_SID = [RAC1] ? RAC1
    [RAC1]$ cd ../psu6-db-12419378/
    psu6-db-12419378[RAC1]$ $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./12419378
    
    ...
    Invoking prereq "checkconflictagainstohwithdetail"
    Prereq "checkConflictAgainstOHWithDetail" passed.
    


  8. The automated patch application process will automatically shutdown and restart all database processes on the node. However, we don't want the automatic restart – because we are applying two PSUs (one for grid and one for database). Disable the instance auto-start for node collabn1 and manually shutdown the instance for patch application.

    collabn1$ srvctl disable instance -d rac -i RAC1
    collabn1$ srvctl stop instance -d rac -i RAC1
    


  9. We will use automated patch application to apply the grid infrastructure PSU. First use oenv to enter the grid environment, then switch to the root user and run opatch auto.

    • Note: any database processes on this node will be automatically shutdown during patching.
    • When prompted for the OCM response file path, enter /home/oracle/ocm.rsp and press enter.
    • When prompted about the grid home being shared, type yes to continue.
    $ oenv
    SIDs here are: grid +ASM1 RAC1
    ORACLE_SID = [RAC1] ? grid
    The Oracle base for ORACLE_HOME=/u01/grid/oracle/product/11.2.0/grid_1 is /u01/app/oracle
    
    [grid]$ cd /mnt/cdrom5/patch/psu2-gi-9655006
    psu2-gi-9655006[grid]$ su
    Password: racattack
    
    psu2-gi-9655006# $ORACLE_HOME/OPatch/opatch auto . -oh $ORACLE_HOME
    Executing /usr/bin/perl /u01/grid/oracle/product/11.2.0/grid_1/OPatch/crs/patch112.pl -patchdir . -patchn . -oh /u01/grid/oracle/product/11.2.0/grid_1 -paramfile /u01/grid/oracle/product/11.2.0/grid_1/crs/install/crsconfig_params
    opatch auto log file location is /u01/grid/oracle/product/11.2.0/grid_1/OPatch/crs/../../cfgtoollogs/opatchauto2011-04-01_22-44-08.log
    Detected Oracle Clusterware install
    Using configuration parameter file: /u01/grid/oracle/product/11.2.0/grid_1/crs/install/crsconfig_params
    OPatch  is bundled with OCM, Enter the absolute OCM response file path:
     /home/oracle/ocm.rsp 
    Can't change permissions of ./.: Read-only file system
    
    Unable to determine if /u01/grid/oracle/product/11.2.0/grid_1 is shared oracle home
    Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no): yes 
    Successfully unlock /u01/grid/oracle/product/11.2.0/grid_1
    patch ././9655006  apply successful for home  /u01/grid/oracle/product/11.2.0/grid_1
    patch ././9654983  apply successful for home  /u01/grid/oracle/product/11.2.0/grid_1
    ACFS-9300: ADVM/ACFS distribution files found.
    ACFS-9312: Existing ADVM/ACFS installation detected.
    ACFS-9314: Removing previous ADVM/ACFS installation.
    ACFS-9315: Previous ADVM/ACFS components successfully removed.
    ACFS-9307: Installing requested ADVM/ACFS software.
    ACFS-9308: Loading installed ADVM/ACFS drivers.
    ACFS-9321: Creating udev for ADVM/ACFS.
    ACFS-9323: Creating module dependencies - this may take some time.
    ACFS-9327: Verifying ADVM/ACFS devices.
    ACFS-9309: ADVM/ACFS installation correctness verified.
    CRS-4123: Oracle High Availability Services has been started.
    


  10. Return to the oracle user and find the Database PSU on the instructor-provided DVD. Use oenv to enter the database environment, then run opatch apply.

    • Run OPatch in local mode so that it does not attempt an automatic rolling install across the entire cluster.
    • When prompted about OCM, leave your username blank and confirm with "y" that you don't want to enter any account information.
    • When prompted whether the local home is ready for patching, type "y" to continue.
    # exit
    
    $ cd ../psu6-db-12419378
    
    psu6-db-12419378$ oenv
    SIDs here are: grid +ASM1 RAC1
    ORACLE_SID = [grid] ? RAC1
    The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1 is /u01/app/oracle
    
    psu6-db-12419378[RAC1]$ cd 12419378
    
    12419378[RAC1]$ $ORACLE_HOME/OPatch/opatch apply -local
    Invoking OPatch 11.2.0.1.5
    
    Oracle Interim Patch Installer version 11.2.0.1.5
    Copyright (c) 2010, Oracle Corporation.  All rights reserved.
    
    Oracle Home       : /u01/app/oracle/product/11.2.0/db_1
    Central Inventory : /u01/app/oraInventory
       from           : /etc/oraInst.loc
    OPatch version    : 11.2.0.1.5
    OUI version       : 11.2.0.1.0
    OUI location      : /u01/app/oracle/product/11.2.0/db_1/oui
    Log file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2011-04-02_16-10-50PM.log
    
    Patch history file: /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch_history.txt
    
    ApplySession applying interim patch '12419378' to OH '/u01/app/oracle/product/11.2.0/db_1'
    
    Running prerequisite checks...
    Patch 12419378: Optional component(s) missing : [ oracle.client, 11.2.0.1.0 ]
    Provide your email address to be informed of security issues, install and
    initiate Oracle Configuration Manager. Easier for you if you use your My
    Oracle Support Email address/User Name.
    Visit http://www.oracle.com/support/policies.html for details.
    Email address/User Name: <enter>
    
    You have not provided an email address for notification of security issues.
    Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
    
    You selected -local option, hence OPatch will patch the local system only.
    
    Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
    (Oracle Home = '/u01/app/oracle/product/11.2.0/db_1')
    
    Is the local system ready for patching? [y|n]
     y
    User Responded with: Y
    Backing up files and inventory (not for auto-rollback) for the Oracle Home
    Backing up files affected by the patch '12419378' for restore. This might take a while...
    Backing up files affected by the patch '12419378' for rollback. This might take a while...
    Execution of 'sh /mnt/cdrom5/patch/psu6-db-12419378/12419378/custom/scripts/pre -apply 12419378 ':
    
    Return Code = 0
    
    Patching component oracle.rdbms.rsf, 11.2.0.1.0...
    ...
    ...
    ...
    ...
    Running make for target itnslsnr
    ApplySession adding interim patch '12419378' to inventory
    
    Verifying the update...
    Inventory check OK: Patch ID 12419378 is registered in Oracle Home inventory with proper meta-data.
    Files check OK: Files from Patch ID 12419378 are present in Oracle Home.
    
    The local system has been patched and can be restarted.
    
    OPatch succeeded.
    


  11. Enable and start the Oracle database instance on node collabn1. After the instance is running, stop and disable the instance on node collabn2. There should be no point at which the database is not running.

    $ srvctl enable instance -d rac -i RAC1
    $ srvctl start instance -d rac -i RAC1
    
    
    $ srvctl disable instance -d rac -i RAC2
    $ srvctl stop instance -d rac -i RAC2
    


  12. Use automated patch application to apply the grid infrastructure PSU to collabn2.

    • Note: any database processes on this node will be automatically shutdown during patching.
    • Copy the OCM response file (/home/oracle/ocm.rsp) to collabn2.
    • When prompted about the grid home being shared, type yes to continue.
    $ scp /home/oracle/ocm.rsp collabn2:/home/oracle/
    $ ssh collabn2
    $ oenv
    SIDs here are: grid +ASM2 RAC2
    ORACLE_SID = [RAC2] ? grid
    The Oracle base for ORACLE_HOME=/u01/grid/oracle/product/11.2.0/grid_1 is /u01/app/oracle
    
    [grid]$ cd /mnt/cdrom5/patch/psu2-gi-9655006
    psu2-gi-9655006[grid]$ su
    Password: racattack
    
    psu2-gi-9655006# $ORACLE_HOME/OPatch/opatch auto . -oh $ORACLE_HOME
    Executing /usr/bin/perl /u01/grid/oracle/product/11.2.0/grid_1/OPatch/crs/patch112.pl -patchdir . -patchn . -oh /u01/grid/oracle/product/11.2.0/grid_1 -paramfile /u01/grid/oracle/product/11.2.0/grid_1/crs/install/crsconfig_params
    opatch auto log file location is /u01/grid/oracle/product/11.2.0/grid_1/OPatch/crs/../../cfgtoollogs/opatchauto2011-04-02_16-42-52.log
    Detected Oracle Clusterware install
    Using configuration parameter file: /u01/grid/oracle/product/11.2.0/grid_1/crs/install/crsconfig_params
    OPatch  is bundled with OCM, Enter the absolute OCM response file path:
     /home/oracle/ocm.rsp 
    Can't change permissions of ./.: Read-only file system
    
    Unable to determine if /u01/grid/oracle/product/11.2.0/grid_1 is shared oracle home
    Enter 'yes' if this is not a shared home or if the prerequiste actions are performed to patch this shared home (yes/no): yes 
    Successfully unlock /u01/grid/oracle/product/11.2.0/grid_1
    patch ././9655006  apply successful for home  /u01/grid/oracle/product/11.2.0/grid_1
    patch ././9654983  apply successful for home  /u01/grid/oracle/product/11.2.0/grid_1
    ACFS-9300: ADVM/ACFS distribution files found.
    ACFS-9312: Existing ADVM/ACFS installation detected.
    ACFS-9314: Removing previous ADVM/ACFS installation.
    ACFS-9315: Previous ADVM/ACFS components successfully removed.
    ACFS-9307: Installing requested ADVM/ACFS software.
    ACFS-9308: Loading installed ADVM/ACFS drivers.
    ACFS-9321: Creating udev for ADVM/ACFS.
    ACFS-9323: Creating module dependencies - this may take some time.
    ACFS-9327: Verifying ADVM/ACFS devices.
    ACFS-9309: ADVM/ACFS installation correctness verified.
    CRS-4123: Oracle High Availability Services has been started.
    


  13. Return to the oracle user and find the Database PSU on the instructor-provided DVD. Use oenv to enter the database environment, then run opatch apply.

    • Run OPatch in local mode so that it does not attempt an automatic rolling install across the entire cluster.
    • When prompted about OCM, leave your username blank and confirm with "y" that you don't want to enter any account information.
    • When prompted whether the local home is ready for patching, type "y" to continue.
    # exit
    
    $ cd ../psu6-db-12419378
    
    psu6-db-12419378$ oenv
    SIDs here are: grid +ASM2 RAC2
    ORACLE_SID = [grid] ? RAC2
    The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1 is /u01/app/oracle
    
    psu6-db-12419378[RAC2]$ cd 12419378
    
    12419378[RAC2]$ $ORACLE_HOME/OPatch/opatch apply -local
    Invoking OPatch 11.2.0.1.5
    
    Oracle Interim Patch Installer version 11.2.0.1.5
    Copyright (c) 2010, Oracle Corporation.  All rights reserved.
    
    Oracle Home       : /u01/app/oracle/product/11.2.0/db_1
    Central Inventory : /u01/app/oraInventory
       from           : /etc/oraInst.loc
    OPatch version    : 11.2.0.1.5
    OUI version       : 11.2.0.1.0
    OUI location      : /u01/app/oracle/product/11.2.0/db_1/oui
    Log file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2011-04-02_17-11-00PM.log
    
    Patch history file: /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch_history.txt
    
    ApplySession applying interim patch '12419378' to OH '/u01/app/oracle/product/11.2.0/db_1'
    
    Running prerequisite checks...
    Patch 12419378: Optional component(s) missing : [ oracle.client, 11.2.0.1.0 ]
    Provide your email address to be informed of security issues, install and
    initiate Oracle Configuration Manager. Easier for you if you use your My
    Oracle Support Email address/User Name.
    Visit http://www.oracle.com/support/policies.html for details.
    Email address/User Name: <enter>
    
    You have not provided an email address for notification of security issues.
    Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  Y
    
    You selected -local option, hence OPatch will patch the local system only.
    
    Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
    (Oracle Home = '/u01/app/oracle/product/11.2.0/db_1')
    
    Is the local system ready for patching? [y|n]
     y
    User Responded with: Y
    Backing up files and inventory (not for auto-rollback) for the Oracle Home
    Backing up files affected by the patch '12419378' for restore. This might take a while...
    Backing up files affected by the patch '12419378' for rollback. This might take a while...
    Execution of 'sh /mnt/cdrom5/patch/psu6-db-12419378/12419378/custom/scripts/pre -apply 12419378 ':
    
    Return Code = 0
    
    Patching component oracle.rdbms.rsf, 11.2.0.1.0...
    ...
    ...
    ...
    ...
    Running make for target itnslsnr
    ApplySession adding interim patch '12419378' to inventory
    
    Verifying the update...
    Inventory check OK: Patch ID 12419378 is registered in Oracle Home inventory with proper meta-data.
    Files check OK: Files from Patch ID 12419378 are present in Oracle Home.
    
    The local system has been patched and can be restarted.
    
    OPatch succeeded.
    


  14. Enable and restart the Oracle database instance on node collabn2.

    $ srvctl enable instance -d rac -i RAC2
    $ srvctl start instance -d rac -i RAC2
    
    $ srvctl status database -d rac
    Instance RAC1 is running on node collabn1
    Instance RAC2 is running on node collabn2
    


  15. Run the catbundle.sql script.

    $ ss
    SQL> @?/rdbms/admin/catbundle.sql psu apply
    
    $ less  /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/catbundle/ catbundle_PSU_RAC_APPLY_2011Apr02_17_28_19.log
    


  16. Optional: if you want more practice working with patches, then try rolling back the database PSU and then try applying it in automated rolling mode (without local flag) or in the “minimum downtime” mode.



Last modified on 5 April 2013, at 05:47