Experimental Physics and
Industrial Control System

"Jeff Hill" <[email protected]> · Thu, 20 Mar 2008 09:00:20 -0600

> Apart from that it would be a more relaxed situation if the ioc would be
> more robust against such wild running clients.

Since clients are allowed to create channels, and subscriptions, at their
own discretion it is difficult to make the IOC robust against a badly
behaved client.  The only solution I can think of would be to implement, per
client, quotas in the access control system - maybe a lot of work.

Jeff

> -----Original Message-----
> From: Schoeneburg, Bernd [mailto:[email protected]]
> Sent: Thursday, March 20, 2008 7:05 AM
> To: Andrew Johnson; Jeff Hill
> Cc: EPICS Tech-Talk
> Subject: Re: Memory Problem related to CA?
> 
> Dear Jeff, dear Andrew,
> thank you very much for analyzing the problem. I have learned a lot
> more. We found out - using host name and port number - which client
> caused the problem. We stopped it for now.
> Apart from that it would be a more relaxed situation if the ioc would be
> more robust against such wild running clients.
> 
> Bernd
> 
> Andrew Johnson schrieb:
> > Jeff Hill wrote:
> >>> What does the first number in parenthesis mean? Something like an
> >>> element count. For this sick connection it was 54! Please look at the
> >> attachment.
> >>
> >> This means that your client has subscribed (called
> >> ca_create_subscription or
> >> ca_add_event) 54 times! One option of course will be to find the
> >> misbehaved
> >> client (the IP address is in the casr report) and convince its author
to
> >> recode his application.
> >
> > One common bug that can cause such a large number of connections is if
> > the channel connection callback routine creates a new subscription
> > whenever a channel connection comes up, even if it already had a
> > subscription from the last time it connected.  The application in
> > question here appears to be the DESY archiver which is likely to have an
> > effect on a large number of IOCs, so I would strongly recommend that
> > this code be reviewed for incorrect channel subscription logic.  A
> > strong indication that this kind of bug is present is if the number
> > increases over an IOC reboot and decreases over an application restart.
> >
> >>> The best
> >>> would be to disconnect all CA connections and give back (at least a
part
> >>> of) the additionally allocated memory back to the system pool.
> >>
> >> We don't currently have an option to do this, but considering that
> >> vxWorks
> >> memory allocation is much better than in the past perhaps one option
that
> >> should be considered for R3.15 is to use free lists dedicated to each
> >> client
> >> that are returned to system pool when the client disconnects.
> >
> > Note that this IOC is running R3.13.10 on vxWorks 5.3.1; if my memory is
> > correct, this is using the old (pre-SENS) TCP/IP stack, so its network
> > buffer management is very different to the SENS stack which has fixed
> > buffer sizes.  The earlier stack could (and did) call malloc() to get
> > new buffers after initialization time.
> >
> > Also, if the memory is being used up by a multiple subscription bug in a
> > client this kind of memory pool rework would make no difference anyway.
> >
> > - Andrew

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System