Experimental Physics and
Industrial Control System

"Jeff Hill" <[email protected]> · Wed, 5 Aug 2009 10:14:39 -0600

Dirk,

Some additional thoughts on your issue.

1) Depending on how your timing system is plumbed (with or without general
time), time-of-day (epicsTime::getCurent) might not advance if the timing
pulses are turned off for a shutdown, and if the time-of-day doesn?t advance
CAC will not expire its timers - this is required for it to send its search
requests.

2) See also mantis 299 and 330 which might or might not be related

PS: In the previous message I meant to say "that part isn?t different
between R3.13 and R3.14"

Jeff
______________________________________________________
Jeffrey O. Hill           Email        [email protected]
LANL MS H820              Voice        505 665 1831
Los Alamos NM 87545 USA   FAX          505 665 5107

Message content: TSPA

> -----Original Message-----
> From: [email protected] [mailto:[email protected]]
> On Behalf Of Jeff Hill
> Sent: Wednesday, August 05, 2009 9:49 AM
> To: 'Dirk Zimoch'; 'tech-talk'
> Subject: RE: CA reconnection problem
> 
> Dirk,
> 
> The R3.14 design is more conservative from a perspective of avoiding
> positive congestion feedback. Here are the primary differences.
> 
> O When a circuit times out the CAC library disconnects the application,
> but
> not the circuit. We rely on TCP to decide when to give up on a circuit,
> and
> disconnect it.
> 
> O When the CAC library sees a beacon anomaly the search rate receives a
> boost, but this boost is only to once every 5 seconds. In contrast, newly
> created channels are initially searched for at a rate which is closer to
> the
> average measured round trip time. The CAC library sends multiple search
> requests per UDP frame, and depending on past success rates also multiple
> UDP frames per periodic search attempt (that part isn?t different between
> R3.14 and R3.15). Nevertheless, if there many channels it can take a
> little
> while to connect all of them if the IOC (or GW in this situation has been
> off for a long time).
> 
> O In R3.14 the CAC library never stops searching for disconnected
> channels,
> but the maximum search rate is adjustable with an environment variable.
> 
> > I cannot see any search requests from that IOC.
> 
> That doesn?t sound right, but bear in mind that if the IOC is sending
> unicasts, and not broadcasts, to the gateway on a switched network then
> the
> network sniffer may or may not see them if it isn?t listening at the
> source
> or the destination cables. Directed broadcasts may also not be seen by the
> network sniffer if it isn?t sniffing directly on the directed broadcast's
> destination subnet.
> 
> The output from dbcar when its interest level argument is set very high
> (say
> to 100) will show many things about the internals of the CAC library.
> Also,
> there is a timer thread created by the CAC library that is scheduling the
> search requests, and you might send a stack trace (if it isn?t simply
> waiting on its time delay event flag).
> 
> Another possibility is that something is amiss in the GW. Does restarting
> the GW improve the situation?
> 
> Jeff
> ______________________________________________________
> Jeffrey O. Hill           Email        [email protected]
> LANL MS H820              Voice        505 665 1831
> Los Alamos NM 87545 USA   FAX          505 665 5107
> 
> Message content: TSPA
> 
> 
> > -----Original Message-----
> > From: [email protected] [mailto:tech-talk-
> [email protected]]
> > On Behalf Of Dirk Zimoch
> > Sent: Wednesday, August 05, 2009 8:09 AM
> > To: tech-talk
> > Subject: CA reconnection problem
> >
> > Hi all (Jeff in particular)
> >
> > I have seen a problem with some of our IOCs today. Several beamline IOCs
> > running
> > 3.14.8 have disconnected CA input links to a central IOC (3.13.10) even
> > though
> > the channels are there. The beamline IOCs run behind ca gateways. It may
> > be that
> > the gateways or the central IOC had been offline for more than some
> hours
> > recently in the last shutdown.
> >
> > Even though the channels are back and I can read them from the beamline
> > network
> > with caget the IOCs don't reconnect.
> >
> > I have started a softioc in the beamline network to generate a beacon
> > anomaly
> > but even then the links do not reconnect. I cannot see any search
> requests
> > from
> > that IOC.
> >
> > Only 3.14.8 systems show this problem. 3.13. systems don't. Is there a
> bug
> > in
> > the 3.14.8 implementation of CAC I should know of?
> >
> > Dirk
> >
> > --
> > Dr. Dirk Zimoch
> > Paul Scherrer Institut, WBGB/006
> > 5232 Villigen PSI, Switzerland
> > Phone +41 56 310 5182

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System