ORACLE.EXE Hangs With Increase in Non-Paged Pool Memory . BEQ Connection may Fail With ORA-12560 (Doc ID 866622.1)

In this Document

APPLIES TO: 

SYMPTOMS

Oracle database instance will hang after several hours . 

The BEQ (bequeath) connection fails with : 

ERROR: 
ORA-12560: TNS:proto adapter error 
ERROR: ORA-12500: TNS:listener failed to start a dedicated server process 

The remote connections fail with : 

TNS-12500: TNS:listener failed to start a dedicated server process 
TNS-12540: TNS:internal limit restriction exceeded 
TNS-12560: TNS:protocol adapter error 
TNS-00510: Internal limit restriction exceeded 
32-bit Windows Error: 8: Exec format error

You may see the below symptoms : 
---------------------------------

1] The SQL> count (*) from v$processes; shows the count below the limit defined in the parameter file. 
2] The memory usage of oracle.exe is well below the 32-bit limitation . 
3] Monitoring the ORACLE.EXE process using process explorer / Task Manager shows an abnormal non-paged pool  count which keeps on increasing. (In terms of MB) 
4] Usually the non-paged pool count for an ORACLE.EXE would be less than an MB. You might see the non-paged pool count for other instance (oracle.exe) on the same machine less than 1MB. 
5] Many handles persist (the process explorer output shown in the screenshot below) thus keeping non-paged pool at a high level .

Even using orastack utility to change the memory usage of the oracle process will not prevent the database 
hang. Using the orastack might delay the occurence of the problem but the instance will eventually hang once the non-paged pool is full. 

For Example: 
-------------
C:\> orastack oracle.exe 500000(every background process will now use 500K instead of 1Mb) 
C:\> orastack tnslsnr.exe 500000(every process connected through SQL*Net will now use 500K instead of 1Mb) 
C:\> orastack sqlplus.exe 500000(every sqlplus connection will now use 500K instead of 1Mb)

Drivers use the NonPaged Pool for many of their requirements because they can be accessed at any interrupt Request Level (IRQL). The IRQL defines the hardware priority at which a processor operates at any given time . The NonPaged Pool cannot be paged out. 

Within the NonPaged pool, you would find handles that are used by applications in the user-mode space as well as Kernel-Mode drivers (typically ending in a .sys file extension). 

what happens to a system when these Pool Resources get depleted? Some of the most common symptoms exhibited are: 

(i) The machine becomes sluggish 
(ii) Users can no longer log on to the machine 
(iii) Console access is sluggish 
(iv) Users cannot connect to file shares or other shared resources 
(v) System hangs including the console itself being unresponsive 

Symptoms such as this are usually the first indication that there is something causing an issue with the machine. 

If the NonPaged Pool on a server has become depleted, the machine will log an Event in the System Log as shown below: 

Event ID 2019 
Event Type: Error 
Event Source: Srv 
Event Category: None 
Event ID: 2019 
Description: The server was unable to allocate from the system NonPaged pool because the pool was empty. 

The following errors typically are an indication that there is no free address space on the server: 

(i) Out-of-memory errors (ORA-4030) 
(ii) Connection spawning errors (ORA-12500) 
(iii) Failure to spawn a process such as a pq slave.

CAUSE

The following are the possible causes for this problem :

1] A faulty application which is causing the handles to increase .

2] A firewall exist between the application and the database server . The applicaton on the  remote client /  server is failing to continue the  connection through the firewall. Connections reach the database but do not seem to return and hence hung sessions linger as orphaned handles.
 

SOLUTION

The following action may help you to fix this problem :

1] Stop the application connecting to this database and monitor the  oracle process for the handle count and the Nonpaged pool size.  If the handle count and the non-paged pool size does not increase drastically then we can confirm that the problem is with the application .

2] Check to see if the firewall exits between the client application and the database sever . Involve your Network/Firewall admin and enusre that there is a 2-way communicaton between the application and the database server.