Wednesday, December 27, 2017

How to Stop Processes in Oracle RAC

When you want to stop an entire node or cluster (for maintenance purposes, for example), you run either the crsctl stop crs command on the node or the crsctl stop cluster -all command, provided you have the required cluster privileges. These commands stop all database instances running on a server or in the cluster and ensure that their state is recovered after you restart the cluster.


Using either of these CRSCTL commands to stop all database instances on a server or in the cluster can lead to the database instances being stopped similar to shutdown abort, which requires an instance recovery on startup. If you use SRVCTL to stop the database instances manually before stopping the cluster, then you can prevent a shutdown abort, but this requires that you manually restart the database instances after restarting Oracle Clusterware.

Since, RAC involves so many components like Database instance, ASM instance, Node application and CRS services, So you need to follow below steps if you don't want to abort your database services and perform manual shutdown of services.

Here I am taking an example of Two node RAC database to perform all operations. Steps followed are as:

1. Shutdown Oracle Home process accessing database.
2. Shutdown RAC Database Instances on all nodes.
3. Shutdown Node applications running on nodes.
4. Shut down the Oracle Clusterware processes and the Oracle ASM instance


1. Shutdown Oracle Home process accessing database :-
There could be oracle and non-Oracle application which access database, So first step is to stop all the applications accessing database. Shut down any processes in the Oracle home on each node that can access a database like Enterprise Manager Grid Control and Database Listener which access database for monitoring and database connections.

[oracle@rac1 ~]$ emctl stop dbconsolc
Stopping Oracle Enterprise Manager………
Stopped…………

[oracle@rac121 ~]$ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): rac121,rac122

[oracle@rac121 ~]$ srvctl stop listener -n rac121

[oracle@rac121 ~]$ srvctl stop listener -n rac122

[oracle@rac121 ~]$ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is not running

2. Shutdown RAC Database Instances on all nodes:-
Shut down all Oracle RAC instances on all nodes. First, I am checking on which server database is running and then stopping and verify for the same.

[oracle@rac121 ~]$ srvctl status database -d racdb
Instance racdb1 is running on node rac121
Instance racdb2 is running on node rac122

[oracle@rac121 ~]$ srvctl stop database -d racdb

[oracle@rac121 ~]$ srvctl status database -d racdb
Instance racdb1 is not running on node rac121
Instance racdb2 is not running on node rac122

We just need to execute one command from any one of the server having database and it will stop all database instances on all servers. If you have more than one database configured on Nodes, then you have to execute this command for each database.
3. Shutdown Node applications running on nodes:-
To stop node applications running on a node, enter the following command, where node is the name of the node where the applications are running. Repeat same command for all nodes one by one. If you face any issue in stopping node applications use "-f" as force option to stop applications.

[oracle@rac121 ~]$ grid_home/srvctl stop nodeapps -n rac121 -f

[oracle@rac121 ~]$ grid_home/srvctl stop nodeapps -n rac122 -f

[oracle@rac121 ~]$ srvctl status nodeapps
VIP rac121-vip.localdomain is enabled
VIP rac121-vip.localdomain is not running
VIP rac122-vip.localdomain is enabled
VIP rac122-vip.localdomain is not running
Network is enabled
Network is not running on node: rac121
Network is not running on node: rac122
ONS is enabled
ONS daemon is not running on node: rac121
ONS daemon is not running on node: rac122 

4. Shut down the Oracle Clusterware processes and the Oracle ASM instance
Shut down the Oracle Clusterware processes and the Oracle ASM instance on each node by entering the "crsctl stop cluster " command on all nodes as the root user.

Note:-To stop Oracle Clusterware and Oracle ASM instances on all nodes in the cluster, you can use a single command: crsctl stop cluster -all.

[oracle@rac121 ~]$ grid_env
[oracle@rac121 ~]$ crsctl check cluster -all
**************************************************************
rac121:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
rac122:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
[root@rac121 ~]# crsctl stop cluster -all
CRS-2673: Attempting to stop 'ora.crsd' on 'rac122'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac122'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac122'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac121'
CRS-2677: Stop of 'ora.DATA.dg' on 'rac122' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac122'
CRS-2677: Stop of 'ora.asm' on 'rac122' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac122' has completed
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac121'
CRS-2673: Attempting to stop 'ora.mgmtdb' on 'rac121'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rac121'
CRS-2677: Stop of 'ora.crsd' on 'rac122' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac122'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac122'
CRS-2673: Attempting to stop 'ora.storage' on 'rac122'
CRS-2677: Stop of 'ora.storage' on 'rac122' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac122'
CRS-2677: Stop of 'ora.ctssd' on 'rac122' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac122' succeeded
CRS-2677: Stop of 'ora.mgmtdb' on 'rac121' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac121'
CRS-2673: Attempting to stop 'ora.MGMTLSNR' on 'rac121'
CRS-2677: Stop of 'ora.DATA.dg' on 'rac121' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac121'
CRS-2677: Stop of 'ora.asm' on 'rac121' succeeded
CRS-2677: Stop of 'ora.MGMTLSNR' on 'rac121' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rac121' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac121' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac121' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac121'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac121'
CRS-2673: Attempting to stop 'ora.storage' on 'rac121'
CRS-2677: Stop of 'ora.storage' on 'rac121' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac121'
CRS-2677: Stop of 'ora.ctssd' on 'rac121' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac121' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac122' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac122'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac122' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac122'
CRS-2677: Stop of 'ora.cssd' on 'rac122' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac121' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac121'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac121' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac121'
CRS-2677: Stop of 'ora.cssd' on 'rac121' succeeded

[root@rac121 ~]# crsctl check cluster -all
**************************************************************
rac121:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
**************************************************************
rac122:
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4530: Communications failure contacting Cluster Synchronization Services daemon
CRS-4534: Cannot communicate with Event Manager
**************************************************************
[root@rac121 ~]# crsctl stop has #run this on each node
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac121'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac121'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac121'
CRS-2673: Attempting to stop 'ora.crf' on 'rac121'
CRS-2677: Stop of 'ora.gpnpd' on 'rac121' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac121' succeeded
CRS-2677: Stop of 'ora.crf' on 'rac121' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac121'
CRS-2677: Stop of 'ora.gipcd' on 'rac121' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac121' has completed
CRS-4133: Oracle High Availability Services has been stopped.

[root@rac121 ~]# crsctl check cluster -all
CRS-4639: Could not contact Oracle High Availability Services
CRS-4000: Command Check failed, or completed with errors.

Note :-
"crsctl stop crs", command will stop Oracle High availability services (OHAS) and Clustware stack in a single command. So we need to run this command on each nodes. From 11g R2, you can do this in two steps, as we showed in above example.

1. To stop Oracle Clusterware and Oracle ASM instances on each nodes in the cluster.
       $crsctl stop cluster #local node
       $crsctl stop cluster –all #all nodes in the cluster

2. Stop Oracle High availability service demon on each node in the cluster.
       $crsctl stop has

Now RAC is completely down. Now you can move ahead with your patching, maintenance activity. Next let's see how to start Oracle RAC cluster database.

Related Links
How to Start Oracle RAC Database

No comments:

Post a Comment