Experimental Physics and
Industrial Control System

"Jeff Hill" <[email protected]> · Wed, 5 Aug 2009 09:49:19 -0600

Dirk,

The R3.14 design is more conservative from a perspective of avoiding
positive congestion feedback. Here are the primary differences. 

O When a circuit times out the CAC library disconnects the application, but
not the circuit. We rely on TCP to decide when to give up on a circuit, and
disconnect it.

O When the CAC library sees a beacon anomaly the search rate receives a
boost, but this boost is only to once every 5 seconds. In contrast, newly
created channels are initially searched for at a rate which is closer to the
average measured round trip time. The CAC library sends multiple search
requests per UDP frame, and depending on past success rates also multiple
UDP frames per periodic search attempt (that part isn?t different between
R3.14 and R3.15). Nevertheless, if there many channels it can take a little
while to connect all of them if the IOC (or GW in this situation has been
off for a long time).

O In R3.14 the CAC library never stops searching for disconnected channels,
but the maximum search rate is adjustable with an environment variable.

> I cannot see any search requests from that IOC.

That doesn?t sound right, but bear in mind that if the IOC is sending
unicasts, and not broadcasts, to the gateway on a switched network then the
network sniffer may or may not see them if it isn?t listening at the source
or the destination cables. Directed broadcasts may also not be seen by the
network sniffer if it isn?t sniffing directly on the directed broadcast's
destination subnet.

The output from dbcar when its interest level argument is set very high (say
to 100) will show many things about the internals of the CAC library. Also,
there is a timer thread created by the CAC library that is scheduling the
search requests, and you might send a stack trace (if it isn?t simply
waiting on its time delay event flag).

Another possibility is that something is amiss in the GW. Does restarting
the GW improve the situation?

Jeff
______________________________________________________
Jeffrey O. Hill           Email        [email protected]
LANL MS H820              Voice        505 665 1831
Los Alamos NM 87545 USA   FAX          505 665 5107

Message content: TSPA

> -----Original Message-----
> From: [email protected] [mailto:[email protected]]
> On Behalf Of Dirk Zimoch
> Sent: Wednesday, August 05, 2009 8:09 AM
> To: tech-talk
> Subject: CA reconnection problem
> 
> Hi all (Jeff in particular)
> 
> I have seen a problem with some of our IOCs today. Several beamline IOCs
> running
> 3.14.8 have disconnected CA input links to a central IOC (3.13.10) even
> though
> the channels are there. The beamline IOCs run behind ca gateways. It may
> be that
> the gateways or the central IOC had been offline for more than some hours
> recently in the last shutdown.
> 
> Even though the channels are back and I can read them from the beamline
> network
> with caget the IOCs don't reconnect.
> 
> I have started a softioc in the beamline network to generate a beacon
> anomaly
> but even then the links do not reconnect. I cannot see any search requests
> from
> that IOC.
> 
> Only 3.14.8 systems show this problem. 3.13. systems don't. Is there a bug
> in
> the 3.14.8 implementation of CAC I should know of?
> 
> Dirk
> 
> --
> Dr. Dirk Zimoch
> Paul Scherrer Institut, WBGB/006
> 5232 Villigen PSI, Switzerland
> Phone +41 56 310 5182

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System