opatchauto - Prerequisite check "CheckActiveFilesAndExecutables" failed.

Opatchauto failed with error.

 Prerequisite check "CheckActiveFilesAndExecutables" failed.

Following active executables are used by opatch process :
 /btkrthdb/db/12.1.0.2/lib/libclntsh.so.12.1

Symptom.

[May 22, 2020 8:09:39 AM] [INFO]    Finish fuser command /sbin/fuser /btkrthdb/db/12.1.0.2/lib/libclntsh.so.12.1 at Fri May 22 08:09:39 UTC 2020
[May 22, 2020 8:09:39 AM] [INFO]    Files in use by a process: /btkrthdb/db/12.1.0.2/lib/libclntsh.so.12.1 PID( 80610 )
[May 22, 2020 8:09:39 AM] [INFO]    Following active executables are not used by opatch process :
                                   
                                   
                                    Following active executables are used by opatch process :
                                    /btkrthdb/db/12.1.0.2/lib/libclntsh.so.12.1
[May 22, 2020 8:09:39 AM] [INFO]    Prerequisite check "CheckActiveFilesAndExecutables" failed.
                                    The details are:
                                   
                                   
                                    Following active executables are not used by opatch process :
                                   
                                   
                                    Following active executables are used by opatch process :
                                    /btkrthdb/db/12.1.0.2/lib/libclntsh.so.12.1
[May 22, 2020 8:09:39 AM] [SEVERE]  OUI-67073:UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.
[May 22, 2020 8:09:39 AM] [INFO]    Finishing UtilSession at Fri May 22 08:09:39 UTC 2020
[May 22, 2020 8:09:39 AM] [INFO]    Log file location: /btkrthdb/db/12.1.0.2/cfgtoollogs/opatchauto/core/opatch/opatch2020-05-22_08-09-07AM_1.log
(END)

-------------------------------------
As per above mentioned log file, there are still some processes using one of the files that is being patched. Here executable libclntsh.so.12.1 is still in use.
Find out which os user and pid is holding.

[obtkrth@indelbtd11 ~]$ fuser /btkrthdb/db/12.1.0.2/lib/libclntsh.so.12.1
/btkrthdb/db/12.1.0.2/lib/libclntsh.so.12.1: 80610m

[obtkrth@indelbtd11 ~]$ lsof /btkrthdb/db/12.1.0.2/lib/libclntsh.so.12.1
COMMAND   PID     USER  FD   TYPE  DEVICE SIZE/OFF    NODE NAME
tnslsnr 80610 obtkrth mem    REG 253,230 58855367 3822203 /btkrthdb/db/12.1.0.2/lib/libclntsh.so.12.1
-------------------------------------


Now Kill this process to proceed with patching.

[obtkrth@indelbtd11 ~]$ kill -9 80610
[obtkrth@indelbtd11 ~]$ lsof /btkrthdb/db/12.1.0.2/lib/libclntsh.so.12.1
[obtkrth@indelbtd11 ~]$

Now no more process, continue with patching.

Monitor SWAP in Linux

create shell script to monitor swap.

#!/bin/bash

cd
. ./.profile

SWAP_THR=70
uname -a |  awk '{print $2}'| read HOST
MAIL_GROUP='vikasthakur232@gmail.com'
df -h swap | awk '{print $5}' | tail -1 | sed 's/.$//'| while read output; do
SWAP_VAR=$(echo $output)
  if [ $SWAP_VAR -gt $SWAP_THR ] ; then
  MESSAGE=${MESSAGE}"Current Swap utilization is $SWAP_VAR percent on $HOME.\n "
  echo -e $MESSAGE | mailx -s "SWAP status on $HOST." $MAIL_GROUP
  fi
done

Non ASM to ASM migration by RMAN

Non ASM to ASM migration by RMAN.

Datafiles, Redo groups and Archive logs are to be moved to ASM. Optional components are controlfile and spfile.
In this doc SID of the db is taken as TEST.
   
Migration of Datafiles, Redo groups and Tempfiles.
Pre steps.
1.    ASM instance should be up and running.
2.    Mount the db using spfile and create a pfile.
3.    Create required Disk groups on ASM instance.
4.                       i. Take full backup of db by RMAN.   – To minimize the downtime go with image copy method.
RMAN> allocate channel c1 device type disk format '+TEST_DATA';
RMAN> backup as copy database tag  'ASM_migration';                              -- Image copy
                                                      RMAN> backup database tag 'ASM_migration';                                           -- Backup set method

ii.    Take incremental backup if you have full backup and strategy of incremental backup.
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Conversion steps.
1.    Take a backup of controlfile and redo to a backup location using cp command.
2.    Update spfile with new value of parameters given below to respective ASM disk group.
Sql> alter system set db_create_file_dest='+TEST_DATA' scope=both;
Sql> alter system set db_create_online_log_dest_1='+TEST_REDO' scope=both;
Sql> alter system set log_archive_dest_1='Location=+TEST_ARCH' scope=both;
3.    Restart the db in mount stage.
4.    Connect db through RMAN.
Below 5, 6, 7 steps are to be performed if the backup is taken as backupset method. Then don’t go with step 8.
5.    Set new location of data files.
RMAN> set newname for datafile 1 to '';   -- do this for all datafiles with their id.
6.    Restore database.
          RMAN> restore database;
7.    Update the control file about the new location of data files.
RMAN> switch datafile all;
Below step is to be performed if the backup is as Image copy method.
8.    Switch database to copy.
         RMAN> switch database to copy;
9.    Recover database.
RMAN> recover database;
10.    Move the temp files to ASM disk group by RMAN.
RMAN> connect target /
RMAN> set newname for tempfile 1 to '+TEST_DATA'; ---- do this for all tempfiles with their id.
RMAN> switch tempfile all;
11.    Open the database.
12.    Move the redo logs to ASM devices by log switch and recreate new log groups with location of ASM disks.

     Migration of Control file.
1.    Restart the db in nomount stage.
2.    Update SPfile to move controlfile to ASM Devices.
     Sql> alter spfile set control_files='+TEST_DATA', scope=spfile;
3.    Restart db in nomount stage.
Sql> shutdown immediate
Sql> startup nomount
4.    Restore control file to ASM disks by RMAN.
                rman target /
RMAN> restore controlfile from <non ASM control file location>; 


Migration of spfile.
1.    Shutdown immediate
2.    Startup mount
3.    Create pfile from spfile;
4.    create spfile='+TEST_DATA' from pfile '$ORACLE_HOME/dbs/inittest.ora';
5.    Login and check the spfile using asmcmd command where ASM instance is running. Location should be something like given below.
+<SID>_DATA/<SID>/PARAMETERFILE/spfile.*****
6.    Make sure we should not have spfile in $ORACLE_HOME/dbs or %ORACLE_HOME%/database location.
7.    Shutdown the database take backup of initfile and remove the contents of initfile and put spfile’s location in pfile.
SPFILE=’+<SID>_DATA/<SID>/PARAMETERFILE/spfile.*****’  in init file.
8.     Startup the db.
9.    Re-link oracle binaries.
10.    Manually verify database parameters,datafiles,redologs,controlfile and spfile location.




This doc is created as per our needs and for my tutorial, If anything goes wrong I am not responsible for the same, before implementing on production or staging be sure to test it on test instance.

Upgrade Grid Infrastructure 11.2.0.4 to 12c PSU 12.1.0.2.160719

1.       Bring down the database instance not the ASM instance.

2.       Create backup of spfile of ASM.

3.       Launch runInstaller

 

 

 

 

Got an issue regarding one of the Kernel Parameter (kernel.panic_on_oops).

Solution – Fix and check again.

 

Before executing rootupgrade.sh –

a)       Patch OPatch with p6880880_121010_Linux-x86-64.zip

b)      Install patch 23273629 to the 12c grid home. GRID Infra PSU 12.1.0.2.160719

Open new Putty Terminal

export ORACLE_HOME=/grid/gi/12.1.0

export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH

cd $ORACLE_HOME/OPatch

opatch apply -oh /grid/gi/12.1.0 -local /shared/23273629/21436941

opatch apply -oh /grid/gi/12.1.0 -local /shared/23273629/23054246

opatch apply -oh /grid/gi/12.1.0 -local /shared/23273629/23054327

opatch apply -oh /grid/gi/12.1.0 -local /shared/23273629/23054341

Once Patching done. Please crosscheck status of ASM instance, if HAS is down bring it UP.

$ sudo /grid/gi/12.1.0/rootupgrade.sh

Hit Next.

Upgrade done successfully. Switch to new terminal, change bash_profile to 12c grid home.

[grid@indelvtdinfdb28 admin]$ crsctl query has softwareversion

Oracle High Availability Services version on the local node is [12.1.0.2.0]

Provide the command.

srvctl config asm

[grid@indelvtdinfdb28 dbs]$ srvctl config asm

ASM home: <CRS home>

PRCA-1057 : Failed to retrieve the password file location used by ASM asm

PRCR-1097 : Resource attribute not found: PWFILE

CAUSE

From 12.1 onwards ASM instance has an attribute called PWFILE(password file). Password file for ASM is not available in 11.2.

This issue is unpublished bug 19449701.  DOC ID 1935891.1

Solution.

[grid@indelvtdinfdb28 dbs]$ crsctl stop resource -all

[grid@indelvtdinfdb28 dbs]$ srvctl remove asm -force 

[grid@indelvtdinfdb28 dbs]$ srvctl add asm -spfile +ORCL_DATA/asm/asmparameterfile/registry.253.917720447 -pwfile $ORACLE_HOME/dbs/orapw+ASM -diskstring '/dev/oracle/*'

[grid@indelvtdinfdb28 dbs]$ srvctl add listener -l listener_ASM -o $ORACLE_HOME -p 26101

[grid@indelvtdinfdb28 dbs]$ srvctl start listener -l listener_ASM

[grid@indelvtdinfdb28 dbs]$ srvctl config asm

ASM home: <CRS home>

Password file: /grid/gi/12.1.0/dbs/orapw+ASM

ASM listener: LISTENER_ASM

Spfile: +ORCL_DATA/asm/asmparameterfile/registry.253.917720447

ASM diskgroup discovery string: /dev/oracle/*

[grid@indelvtdinfdb28 dbs]$ srvctl start asm

[grid@indelvtdinfdb28 dbs]$ crsctl stop has

[grid@indelvtdinfdb28 dbs]$ crsctl start has

[grid@indelvtdinfdb28 dbs]$ crsctl start <diskgroup> -unsupported