EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  <20092010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  <20092010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RE: Very slow reconnection to medm after IOC reboot
From: "Jeff Hill" <[email protected]>
To: "'Mark Rivers'" <[email protected]>, "'Andrew Johnson'" <[email protected]>
Cc: "'epics'" <[email protected]>
Date: Fri, 23 Jan 2009 11:27:14 -0700
Mark,

How many channels were involved in this disconnect / reconnect scenario?

> Stop IOC, wait 10 seconds, restart IOC.
> Time for medm to reconnect all channels: less than 10 seconds.

> Stop IOC, wait 15 minutes, restart IOC.
> Time for medm to reconnect all channels: 4 minutes

When the CA library disconnects from the IOC it starts looking for the
consequently disconnected channels with a fast search interval, but if the
library isn't immediately successful it exponentially backs off to a much
lower rate. We want that behavior in order to reduce the load on the
network.

If, after 15 minutes, the IOC comes up and the CA client library manages to
find some of its disconnected channels it has no way of knowing that the
balance of its remaining disconnected channels are on this new IOC. Consider
a worst case scenario; what if this particular CA client had a very large
number of disconnected channels. From my perspective, it wouldn't be a good
idea to search at the highest rate for these channels just because a new IOC
was seen. The client library does however adopt a compromise design where it
searches at an intermediate rate if it sees a beacon anomaly, and depending
on how many channels are disconnected this intermediate rate might result in
a 4 minute reconnect time. Furthermore, I think I recall correctly that the
event of connecting to a new IOC will also cause this same intermediate
search rate boost that is granted when a beacon anomaly event is detected.

Part of the rationale behind the current design is that if it was necessary
to wait a long time for the IOC to come up then waiting as long as the
number of channels divided by the number of channels per search attempt
times this intermediate delay (maybe as much as 4 minutes in this situation)
for the channels to reconnect isn't going to shatter our world. It is
particularly important to avoid designs which cause unstable traffic
insertion feedback when there is network congestion.

Jeff


> -----Original Message-----
> From: Mark Rivers [mailto:[email protected]]
> Sent: Thursday, January 22, 2009 3:28 PM
> To: Andrew Johnson
> Cc: Jeff Hill; epics
> Subject: RE: Very slow reconnection to medm after IOC reboot
> 
> Hi Andrew,
> 
> Thanks for the response.
> 
> Here is the output of all env settings that start with the string EPICS,
> before I manually set EPICS_CAS_BEACON_ADDR_LIST:
> 
> $ printenv | grep EPICS
> EPICS_CA_AUTO_ADDR_LIST=NO
> EPICS_HOST_ARCH=win32-x86
> EPICS_CA_ADDR_LIST=localhost 164.54.160.255
> EPICS_DISPLAY_PATH=C:\EPICS\adls\
> EPICS_CA_MAX_ARRAY_BYTES=10000000
> 
> If I run with just those env settings I get the following message when I
> boot the IOC:
> 
> The CA server's beacon address list was empty after initialization?
> 
> If I add the env you suggested so they now look like this:
> 
> $ printenv | grep EPICS
> EPICS_CA_AUTO_ADDR_LIST=NO
> EPICS_CAS_BEACON_ADDR_LIST=164.54.160.255
> EPICS_HOST_ARCH=win32-x86
> EPICS_CA_ADDR_LIST=localhost 164.54.160.255
> EPICS_DISPLAY_PATH=C:\EPICS\adls\
> EPICS_CA_MAX_ARRAY_BYTES=10000000
> 
> Then when I start the IOC I don't get the message about the empty beacon
> address list.
> 
> This is a new problem with 3.14.10.  I run identical IOCs built with
> 3.14.8.2 and I don't get that error message with exactly the same
> environment settings.
> 
> However, even though the error message is gone the performance is only
> marginally improved, if at all.
> 
> Here is the result with EPICS_CAS_BEACON_ADDR_LIST set:
> 
> ******************************
> Windows IOC (3.14.10), Windows medm client (built with 3.14.9) on same
> PC.
> 
> Stop IOC, wait 10 seconds, restart IOC.
> Time for medm to reconnect all channels: less than 10 seconds.
> 
> Stop IOC, wait 15 minutes, restart IOC.
> Time for medm to reconnect first channel: 10 seconds
> Time for medm to reconnect all channels: 4 minutes
> 
> 
> The differences from my previous test (when I had the CA server beacon
> address error):
> 
> The time for the very first channel to reconnect was reduced from 60
> seconds to 10 seconds.  I am not sure how reproducible this is.
> 
> The time for all channels to connect was reduced from 4 minutes and 25
> seconds to 4 minutes.  Again, this may not be a statistically
> significant improvement.
> 
> Thus, even with EPICS_CAS_BEACON_ADDR_LIST set the performance is really
> bad.  And I am not the only one seeing it, Lewis Muir reported similar
> long reconnection times earlier this afternoon.
> 
> Mark
> 
> 
> 
> -----Original Message-----
> From: Andrew Johnson [mailto:[email protected]]
> Sent: Thursday, January 22, 2009 3:47 PM
> To: Mark Rivers
> Cc: Jeff Hill; epics
> Subject: Re: Very slow reconnection to medm after IOC reboot
> 
> On Thursday 22 January 2009 15:32:47 Mark Rivers wrote:
> > I noticed that when I start win32-x86 IOCs built with 3.14.10 I get
> the
> > following message just after iocInit:
> >
> > *******
> > The CA server's beacon address list was empty after initialization?
> > *******
> >
> > I do not get this message when running the identical IOC built with
> > 3.14.8.2.
> >
> > Is this significant?  Why am I getting this message?
> 
> Highly, it means that the IOC is not sending out any CA beacons at all,
> which
> explains why the clients are not reconnecting to it immediately.  Try
> setting
> EPICS_CAS_BEACON_ADDR_LIST on the IOC to your 164.54.160.255 broadcast
> address, although it should default to the value of EPICS_CA_ADDR_LIST
> if not
> set so I don't understand why it's ignoring that (unless you have other
> env
> settings for the IOC that you haven't told us about).
> 
> - Andrew
> --
> The best FOSS code is written to be read by other humans -- Harold Welte


Replies:
RE: Very slow reconnection to medm after IOC reboot Mark Rivers
References:
Very slow reconnection to medm after IOC reboot Mark Rivers
RE: Very slow reconnection to medm after IOC reboot Mark Rivers
Re: Very slow reconnection to medm after IOC reboot Andrew Johnson
RE: Very slow reconnection to medm after IOC reboot Mark Rivers

Navigate by Date:
Prev: Re: RTEMS soft reboot Kate Feng
Next: RE: Very slow reconnection to medm after IOC reboot Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  <20092010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: RE: Very slow reconnection to medm after IOC reboot Mark Rivers
Next: RE: Very slow reconnection to medm after IOC reboot Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  <20092010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 31 Jan 2014 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·