Experimental Physics and
Industrial Control System

Andrew Johnson <[email protected]> · Fri, 5 Apr 2013 10:00:16 -0500

Hi Benjamin,

On 2013-04-05 Benjamin Franksen wrote:
> I have a strange problem with an IOC runnign EPICS base 3.14.8. Half an
>  hour ago, it suddenly started issuing error messages like
> 
> ../online_notify.c: CA beacon (send to "193.149.12.255:5065") error was
> "S_errno_ENOBUFS"
> 
> which I now get periodically (every few seconds).
> 
> Now, what is funny is that the IOC does not have any records on it. I have
> a very similar IOC in the same network (same base version) that /does/
> have records on it and which does /not/ issue these messages.

The ENOBUFS error is originating in the network stack, not in CA.  The absence 
of records doesn't prevent iocInit() from starting the task which sends out 
beacons and is generating this message.

> There is probably some rogue CA client on the network that broadcasts name
> resolution requests with a (too) high frequency (this is our development
> network, not operation, fortunately), but I have not yet managed to find
>  out its IP address.

Nope, that's not the issue at all.  The problem is that RSRV is getting an 
error from the network layer whenever it tries to send a beacon, which happens 
every 15 seconds on the IOC.

> "casr 2" says
> 
> Channel Access Server V4.11
> No clients connected.
> UDP Server:
> UDP 193.149.12.6:38238(): User="", V4.11, 0 Channels, Priority=0
>         Task Id=0x1d31698, Socket FD=14
>         Secs since last send 2853.75, Secs since last receive 1550.10
>         Unprocessed request bytes=16, Undelivered response bytes=0
>         State=up
>         168 bytes allocated
> 
> There are currently 1176 bytes on the server's free list
> 7 client(s), 0 channel(s), 0 event(s) (monitors) 0 putNotify(s)
> 0 small buffers (16384 bytes ea), and 0 jumbo buffers (16408 bytes ea)
> The server's resource id conversion table:
> Bucket entries in use = 0 bytes in use = 16404
> Bucket entries/hash id - mean = 0.000000 std dev = 0.000000 max = 0
> The server's array size limit is 16408 bytes max
> Channel Access Address List
> 193.149.12.255:5065
> 
> 
> This is strange in and of itself, since it first says "No clients
>  connected" (which I expected, since there are no records) and then later
>  "7 client(s),..." which I find disturbing.

You're reading the error message wrong, it's telling you that there is space 
available for 7 clients from the server's free list.

> So, if anyone has an idea what might be going on here I'd be glad for an
> explanation. BTW, I restarted the IOC which silences the error messages for
> some time (a few minutes, maybe) after which they start again.
> 
> How do I find rogue CA clients (if that causes the problem at all, maybe
> something else is wrong)? Could network problems cause this? Is there maybe
> a known bug in 3.14.8 that could explain this?

You should run an inetstatShow and look at the size of the queues for all of 
the connected sockets that is reports.  Hopefully you'll find a socket which 
has lots of data queued, and that will give you a hit where to look further.  
You should also run netStackSysPoolShow and netStackDataPoolShow (or mbufShow 
if you're on a *really* old vxWorks version) which tell you how many network 
buffers of each size the stack has available. and also give you some other 
information about how often it has failed to find a large enough buffer.

HTH,

- Andrew
-- 
There has grown up in the minds of certain groups in this country the
notion that because a man or corporation has made a profit out of the
public for a number of years, the government and the courts are charged
with the duty of guaranteeing such profit in the future, even in the face
of changing circumstances and contrary to public interest. This strange
doctrine is not supported by statute or common law. Neither individuals
nor corporations have any right to come into court and ask that the clock
of history be stopped, or turned back. -- Robert A Heinlein, Life-Line

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System