In this Document

Symptoms

Changes

Cause

Solution

Scalability RAC Community

References

---

 

APPLIES TO: 

Oracle Database - Enterprise Edition - Version 11.2.0.2 and later
Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Oracle Database Cloud Exadata Service - Version N/A and later
Information in this document applies to any platform.
This issue impacts environments that do not have multicast enabled for the private network in the following situations:

New installations of Oracle Grid Infrastructure 11.2.0.2 where multicast is not enabled on 230.0.1.0
Upgrades to Oracle Grid Infrastructure 11.2.0.2 from a pre-11.2.0.2 release where multicast is not enabled on 230.0.1.0 or 224.0.0.251
Installation of GI PSU 11.2.0.3.5, 11.2.0.3.6, 11.2.0.3.7 where multicast is not enabled on 230.0.1.0 or 224.0.0.251
Installation or upgrade to 12.1.0.1.0 where multicast is not enabled on 230.0.1.0 or 224.0.0.251

 

SYMPTOMS

If multicast based communication is not enabled as required either on the nodes of the cluster or on the network switches used for the private interconnect, the root.sh, which is called as part of a fresh installation of Oracle Grid Infrastructure 11.2.0.2, or the rootupgrade.sh (called as part of an upgrade to Oracle Grid Infrastructure 11.2.0.2) will only succeed on the first node of the cluster, but will fail on subsequent nodes with the symptoms shown below:

 

 

Symptom verification 
To verify that Oracle CSS daemon fails to start in clustered mode due to a multicasting issue on the network, the ocssd.log file (located under $GI_HOME/log/<nodename>/cssd/ocssd.log) must be reviewed. In case, joining the cluster fails because of such an issue, the following can be observed: 

1. When CSS starts in clustered mode to join an existing cluster, we will see an entry in the CSSD log indicating that CSS will attempt to establish communication with a peer in the cluster. For this analysis, we see in the CSSD log for <node2> that communication is attempted with <node1>, which looks similar to:  

 

2.  Shortly after the above log entry we will see an attempt to establish communication to <node1> from <node2> via multicast address 230.0.1.0, port 42424 on the private interconnect (here: 192.168.x.x):

3.  If the communication can be established successfully, we will see a log entry on node2 containing "gipchaLowerProcessAcks: ESTABLISH finished" for the peer node (<node1>). If the communication cannot be established, we will not see this log entry. Instead, we will see an entry indicating that the network communication cannot be established. This entry will look similar to the one shown below:

The above log entry indicates that CSSD is unable to establish network communication on the interface used for the private interconnect. In this particular case, the issue was that multicast communication on the 230.0.1.0 IP was blocked on the network used as the private interconnect.

 

 

CHANGES

• New installations of Oracle Grid Infrastructure 11.2.0.2
• Upgrades of a previous release to Oracle Grid Infrastructure 11.2.0.2
• Installation of the 11.2.0.3.5, 11.2.0.3.6, 11.2.0.3.7 GI PSUs where multicast is not enabled on the 230.0.1.0 or 224.0.0.251 multicast addresses
• New installations of Oracle Grid Infrastructure 12.1.0.1 where multicast is not enabled on the 230.0.1.0 or 224.0.0.251 multicast addresses
• Upgrades of a previous release to Oracle Grid Infrastructure 12.1.0.1 where multicast is not enabled on the 230.0.1.0 or 224.0.0.251 multicast addresses

 

CAUSE

Assuming that Cluster Verify (cluvfy) has succeeded regarding the network checks on all nodes of the cluster or the symptoms described above are observed as part of an upgrade to Oracle Grid Infrastructure 11.2.0.2 (which means that the current release does not encounter such communication issues), this issue is probably caused by multicast not being enabled on the network used as the private interconnect.

Background information for 11.2.0.2
Oracle Grid Infrastructure 11.2.0.2 introduces a new feature called "Redundant Interconnect Usage", which provides an Oracle internal mechanism to make use of physically redundant network interfaces for the Oracle (private) interconnect. As part of this new feature, multicast based communication on the private interconnect is utilized to establish communication with peers in the cluster on each startup of the stack on a node. Once the connection with the peers in the cluster has been established, the communication is switched back to unicast. Per default, the 230.0.1.0 address (port 42424) on the private interconnect network is used for multicasting. Another IP can be enabled using the patch mentioned below, if it is determined that using the 230.0.1.0 IP causes the multicast communication to fail. Multicasting on either of these IPs and the respective port must, however, be enabled and functioning across the network and on each node meant to be part of the cluster. If multicasting is not enabled as required, nodes will fail to join the cluster with the symptoms discussed.

Background information for 11.2.0.3.5, 11.2.0.3.6, 11.2.0.3.7 GI PSUs and 12.1.0.1
With 11.2.0.3 GI was enhanced to utilize broadcast or multicast (on 230.0.1.0 or 224.0.0.251 addresses) to bootstrap. However the 11.2.0.3.5 GI PSU introcuces a new issue with effectivly disables the broadcast functionality (Bug 16547309).  Do note that most networks do support multicast on the 224.0.0.251 multicast address without any special configuration, therefore the odds of this being an issue for 11.2.0.3.5 - 11.2.0.3.7 and 12.1.0.1 are greatly reduced.

 

Configuring various network switches for multicast
While the most common reason for multicast based communication to fail with Oracle Grid Infrastructure 11.2.0.2 is probably the 230.0.1.0 multicast address being used by default, multicast being disabled or misconfigured on the network switches is an equally common cause. The mcasttest.pl-tool provided as part of this note will give you an indication whether or not multicasting has been disabled on the network switches, if it fails to use multicast on either the 230.0.1.0 or 224.0.0.251 multicast address for example. In general, work with your network administrator, if you suspect multicasting has been disabled on the network switches, affecting the private interconnect communication accordingly.

Cisco Nexus Switches - general 
The Cisco Nexus switches can be complicated to configure with multicasting. If multicasting cannot be properly configured, then these switches have the ability to turn off IGMP Snooping on a per-VLAN basis. This has the effect of having the switch handle multicasts like it would a broadcast. Since the network used for the interconnect is supposed to be private (where only the members of the cluster have access to this network), broadcasting every multicast packet to every NIC on the VLAN should pose no problem. The command to turn off IGMP Snooping on a VLAN is:

Cisco Catalyst 
Turning off IGMP Snooping on a Cisco Catalyst switch is a more complex process, which is not covered in this note. For the Catalyst family of switches, IGMP Snooping can only be turned off globally and not on a per-VLAN basis. Consult the Cisco Catalyst manuals for how to properly configure multicasting for the switches over-all usage. Note that turning off IGMP Snooping on the Catalyst may cause other systems or programs to fail.

Cisco Nexus 7000 Switch 
A feature of the Cisco Nexus 7000 switch is a filter that effects large UDP messages. This command is designed to filter IP packets with the "do not fragment" bit set along with a non-zero "Fragment offset". Oracle RAC will send UDP messages that exceed the carrying capacity of a standard UDP packet. The IP layer sees this and sends the left-over data in IP Fragments. These IP Fragments will then have the "do not fragment" bit set along with a non-zero "Fragment offset". The switch will not deliver these packets as they are filtered out by this rule. 

To validate this issue, perform a tcpdump (with the -s 0 flag to capture the entire packet) on each node during the Oracle Clusterware installation at the time the root.sh script is run. The root.sh command will start the Oracle cluster services on the first node. When it is run on the second node, large UDP packets can occur, which will generate the IP Fragments that will be dropped. When one reads the two tcpdump files, on will see an IP Fragment packet leave the second node, yet it will not arrive on the first node. The first node will send out ICMP "Fragment reassembly time exceeded" packets as a result. To turn this rule off issue the following command:

These settings apply only to the Cisco Nexus 7000 switch. This command may be available on other models of the Nexus switch, but not all of the Nexus models support it. Check the Cisco manual for the model of concern.

 

 

SOLUTION

For 11.2.0.2 Installations

It has been found that using a 230.0.1.0, port 42424, network address for multicasting can be problematic with some network configurations. Therefore, Oracle has released Patch: 9974223 on top of Oracle Grid Infrastructure 11.2.0.2. This patch enables multicasting on the 224.0.0.251 multicast address (port 42424) in addition to the 230.0.1.0 (port 42424) address used by default. Multicast must generally be enabled on one of these two addresses to allow Oracle Grid Infrastructure to successfully start on all nodes configured to join a particular cluster.

For 11.2.0.3.5, 11.2.0.3.6 and 11.2.0.3.7 GI PSU  and 12.1.0.1 Installations

11.2.0.3.5 - 11.2.0.3.7 GI PSU and 12.1.0.1 installations will ONLY be impacted if those installations were relying on the broadcast functionality that was initially introduced in 11.2.0.3 base.  For these installations the most simplistic solution is to enable multicast on either the 230.0.1.0 or 224.0.0.251 address (only 1 of the 2 addresses needs to be functional).  For those installations that are unable to enable multicast, Patch 16547309 is available for some platforms to re-enable broadcast functionality.  If you require this patch (due to not being able to enable multicast) and it is not available for your version/platofrm please raise an SR with support to request the fix.  The fix for Bug 16547309 will be included in the 11.2.0.3.8 and 12.1.0.1.2 GI PSUs.

For 12.2.0.1 Installations

"Multicasting is required on the private interconnect. For this reason, at a minimum, you must enable multicasting for the cluster for the following..."

Reference:  http://docs.oracle.com/database/122/CWLIN/multicast-requirements-for-networks-used-by-oracle-grid-infrastructure.htm#CWLIN-GUID-09DA3502-5880-4E72-88C0-CBE8E4893DF7

 

Problem validation
Before applying Patch: 9974223, Patch 16547309 or any subsequent Oracle Grid Infrastructure (GI) Bundle Patch including which include the mentioned fixes, understand that the issue described in this note does not apply under the following circumstances:

 If you have already run the root.sh (rootupgrade.sh) on any node of the cluster:

Multicast can be vaildated using the mcasttest.pl tool attached to this note to validate whether or not multicast on the 230.0.1.0 or 224.0.0.251 address can be used on the private interconnect interface(s).  CLUFFY in 11.2.0.3 and above does contain similar checks but will report pass if the broadcast check passes, with mentioned issue which breaks broadcast in 11.2.0.3.5 - 11.2.0.3.7 GI PSU and 12.1.0.1 this pass is a false positive.  For this reason it is recommended to use mcasttest.pl program to validate multicast functionality before patching/upgrading/installing the mentioned releases.

How to use the mcasttest-tool 

• Download the mcasttest.pl perl program from this note and extract the contents of the package to node 1 of your cluster:

• The mcasttest.pl program can be found in the mcasttest directory extracted in the previous step.
• The mcasttest.pl program requires two arguments:
  1. The node list (specified with -n)
  2. The list of interfaces to be used for the private interconnect (specified with -i).
• The mcasttest is only to be executed on a single node in the cluster. It will use password-less SSH based access (also required for the Grid Infrastructure software installation) to distribute 
  and execute the program on the remote nodes. The command string to execute mcasttest is:

The example below tests multicast for a two node cluster (<node1> and <node2>), in 
which two network interfaces (eth1 and eth2) are to be used for the private interconnect:

As shown in the example above, the test has failed for the 230.0.1.0 address, but succeeded for the 224.0.0.251 multicast address. In this case, Patch: 9974223 must be applied to enable Oracle Grid Infrastructure to use the 224.0.0.251 multicast address.  

Interpreting the outcome of the mcasttest-tool correctly:

• Should the mcasttest.pl test-tool have failed for both, the 230.0.1.0 and the 224.0.0.251 address
  • you must work with your System and / or Network Administrator to enable multicast on one of the addresses.
• Should the mcasttest.pl test-tool have failed for the 230.0.1.0 address only
  • Apply Patch: 9974223 or any subsequent GI PSU (fix provided GI PSU 1 and above)
• Should the mcasttest.pl test-tool have returned "success" for both, the 230.0.1.0 and the 224.0.0.251 address
  • No specific patch application is required and you can proceed with the installation.  

When and how to patch for 11.2.0.2
Ideally, Patch: 9974223 or the latest 11.2.0.2 GI PSU  (fix included in GI PSU 1 and above) is applied before any root.sh or rootupgrade.sh has ever been run on any of the nodes meant to form a cluster. It is highly recommended that the latest 11.2.0.2 GI PSU is installed to obtain the fix for Bug: 9974223 (instead of Patch: 9974223 standalone). In order to apply Patch: 9974223 either directly or as part of a Bundle Patch, perform the following steps:

• Perform an Oracle Grid Infrastructure 11.2.0.2 installation or upgrade
• Right before the first root.sh (or rootupgrade.sh) is supposed to be run, leave the current installation behind.
  • Do not close the installer or abort an operation in progress.
  • Leave the current installation as-is and open a new terminal.
• Download and prepare the application of Patch: 9974223 either directly or as part of a GI Bundle.
• Unlike described in the patch readme (of the GI Bundle Patch),
  • do not use "opatch auto"
  • Instead, use: "opatch napply -local"
    • IF this is a fresh installation, you do not need to run rootcrs.pl -unlock and rootcrs.pl -patch, because root.sh is not run yet, so all the files are unlocked.

• After you have patched every node in the cluster, return to the original installation
• Proceed to run the root.sh (rootupgrade.sh) on all nodes and follow the instructions on the screen.

When and how to patch for 11.2.0.3.5, 11.2.0.3.6, 11.2.0.3.7

As previously mentioned, the most simplistic solution is to enable multicast on either the 230.0.1.0 or 224.0.0.251 address to address.  If this is not possible within your environment, the recommended solution is to install the fix for Bug 16547309prior to executing "rootcrs.pl -patch".

When and how to patch for 12.1.0.1

Again, the most simplistic solution is to enable multicast on either the 230.0.1.0 or 224.0.0.251 address to address this issue.  If this is not possible within your environment, the recommended solution is to install the fix for Bug 16547309 or the latest GI PSU containing the fix (12.1.0.1.2) prior to running root.sh or rootupgrade.sh.  In order to apply Patch 16547309 prior to the execution of root.sh or rootupgrade.sh, perform the following steps:

• Perform an Oracle Grid Infrastructure 12.1.0.1 installation or upgrade
• Right before the first root.sh (or rootupgrade.sh) is supposed to be run, leave the current installation behind.
  • Do not close the installer or abort an operation in progress.
  • Leave the current installation as-is and open a new terminal.
• Download and prepare the application of Patch 16547309 either directly or as part of a GI Bundle.
• Unlike described in the patch readme (of the GI Bundle Patch),
  • do not use "opatch auto"
  • Instead, use: "opatch napply -local"
    • IF this is a fresh installation, you do not need to run rootcrs.pl -unlock and rootcrs.pl -patch, because root.sh is not run yet, so all the files are unlocked.

 

 

Scalability RAC Community

To discuss this topic further with Oracle experts and industry peers, we encourage you to review, join or start a discussion in the My Oracle Support Scalability RAC Community.

REFERENCES