Experimental Physics and
Industrial Control System

"Lawrence T. Hoff" <[email protected]> · Fri, 28 Jan 2005 21:37:02 -0500

	This issue is not specific to EPICS, we noticed it in a
non-EPICS VxWorks system at RHIC. However, since EPICS is
commonly run on VxWorks, this problem could show up on an
EPICS system. I mention this just in case someone in the
community has run into it........

	The problem is a bug in the NFS client software in
VxWorks. The S/W does not gracefully handle the case when
"user" MBUFS have been depleted, but "system" MBUFs have
not. Under these circumstances, it is possible to create 
a socket, but may not be possible to bind that socket to 
a local port. This causes the NFS client S/W to get confused,
and neglect to close the socket. The socket (and its 
associated file descriptor) are forever orphaned (until a 
reboot).

	The situation at RHIC where this occurred was a system
which provided data to a user display related to superconducting
magnet quench detection, and optionally stored "trend" data 
directly on a (Windows NT) file server via NFS. Under certain 
circumstances (enough simultaneous operator displays combined 
with a network interruption), MBUFS could temporarily become 
depleted. Normally this was no big deal. After the network
interruption was corrected, the MBUFs were freed. However, if
the data logging was enabled during that time, each attempted
NFS operation resulted in one "leaked" socket and file descriptor.
In extreme cases, all 256 file descriptors were leaked, the 
system required a reboot and the trend data was lost.

	I was able to substitute a "fixed" clnt_udp.c module,
based on the source code provided in the "unsupported"
directory, which cures this problem. I am currently working
with WRS on incorporating this fix in future releases of
VxWorks. In the meanwhile, if anyone needs a fixed NFS client,
contact me, and I can send the source code.

	A clear signature that this has happened on a VxWorks
system is one or more UDP sockets which are not bound to
any address. These can be seen via inetstatShow:

f02624   UDP        0      0  0.0.0.0.0             0.0.0.0.0
f02498   UDP        0      0  0.0.0.0.0             0.0.0.0.0

HTH -- Larry

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System