Experimental Physics and
Industrial Control System

"Jeff Hill" <[email protected]> · Wed, 29 Jan 2003 15:36:15 -0700

All,

I am just now adding the following paragraph to the CA reference
manual. Any comments or revisions are appreciated.

Jeff

Problems with "mbufs" on vxWorks

Many Berkley UNIX derived Internet Protocol (IP) kernels use a
memory management scheme with a fixed sized low level memory
allocation quantum called an "mbuf". When "S_errno_ENOBUFS"
messages are emitted on vxWorks systems this indicates that your
vxWorks IP kernel is running low on mbuf buffers. The fix is to
reconfigure vxWorks to include a larger IP kernel mbuf pool. The
procedure for adjusting the size of the mbuf pool varies
depending on the version of vxWorks, so consult your vxWorks
documentation. In certain earlier versions of vxWorks, the IP
kernel does not appear to recover well if it gets into an mbuf
starvation situation.  

There can be contributing circumstances. 

The total number of connected clients is high. Each active socket
requires dedicated mbufs for protocol control blocks, and for any
data that might be pending in the operating system for
transmission to Channel Access or to the network at a given
instant. If you increase the vxWorks limit on the maximum number
of file descriptors then it may also be necessary to increase the
size of the mbuf pool.

The server has multiple connections where the server's sustained
event (monitor subscription) production rate is higher than the
client's or the network's sustained event consumption rate. This
ties up a per socket quota of mbufs for data that are pending
transmission to the the client via the network.

The server does not get a chance to run (because some other
higher priority thread is running) and the CA clients are sending
a high volume of data over TCP or UDP. This ties up a quota of
mbufs for each socket in the server that isn't being reduced by
the server's socket read system calls.

The server has multiple stale connections. Stale connections
occur when a client is abruptly turned off or disconnected from
the network, and an internal "keepalive" timer has not yet
expired for the virtual circuit in the operating system, and
therefore mbufs may be dedicated to unused virtual circuits. This
situation is made worse if there are active monitor subscriptions
associated with stale connections which will rapidly increase the
number of dedicated mbufs to the quota available for each
circuit.

The EPICS command "casr" shows how many CA clients are connected.
The vxWorks command "inetstatShow" indicates how many bytes are
pending in mbufs and indirectly (based on the number of circuits
listed) mbuf based protocol control blocks. The vxWorks commands
(availability depending on vxWorks version) mbufShow,
netStackSysPoolShow, and netStackDataPoolShow indicate how much
space remains in the mbuf pool.   

__________________________________________________________
Jeffrey O. Hill               Mail         [email protected]
LANL MS H820                  Voice        505 665 1831
Los Alamos NM 87545 USA       Fax          505 665 5107

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System