EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  <20092010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  <20092010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RE: Very slow reconnection to medm after IOC reboot
From: "Jeff Hill" <[email protected]>
To: "'Mark Rivers'" <[email protected]>, "'Andrew Johnson'" <[email protected]>
Cc: "'epics'" <[email protected]>
Date: Fri, 23 Jan 2009 15:40:22 -0700
> But in this case I am doing development on a
> single PC, running a local client connected to channels on that same PC,
> but I have to wait 4 minutes for the channels to reconnect just because
> I rebuilt the application.  This is not acceptable.

Perhaps I am a weasel to look only at this usage case in isolation, but it
seems that this might be what happened.

O stopping the IOC
O rebuilding application
O maybe other stuff that takes a long time 
O starting the IOC

But perhaps this would work just as well (with very fast reconnect times)

O rebuild the application
O reboot the IOC

Nevertheless, if by chance one ends up with the first scenario, restarting
the MEDM screen isn't, from my perspective, exactly heavy lifting -
considering that we are presumably waiting at the edge of the seat for the
channel's to reconnect. Synchronization might be the primary aggravation,
but perhaps it will even be successful the first try if one waits
approximately long enough for the IOC to finish rebooting before restarting
the medm screen.

> It seems like the client library should know what IOC previously hosted
> the PVs that are now disconnected.  If that same IOC now comes back up,
> it seems like a reasonable assumption that the same IOC is going to be
> hosting those channels. So doing a high-rate search for those channels
> has a very high liklihood of success?
> 	

The code doesn't currently do that. It seems that this might be a reasonable
idea to add to the list for R3.15. The primary negative would be storage
overhead for keeping the network address of the IOC last connected to with
every channel. It's preferable to keep the network address in circuit
specific snap-in since network addresses are circuit specific. Perhaps we
could create a special type of pseudo circuit (satisfying the polymorphic
circuit interface) for keeping track of last-connected-to state, and store a
pointer to such an instance with the channel when it disconnects. Another
negative of course is complexity, and maybe also CPU overhead for running
the list of channels and checking each one of them for a match with the new
IOC's address.

> Is there an environment variable I can set to change the search rate or
> algorithm?  I agree that in general we want to avoid anything that can
> generate network storms.  

You can set EPICS_CA_MAX_SEARCH_PERIOD, but the code imposes, currently, a
lower limit of 60 seconds. So this isn't probably what you want, but it is
maybe what your EPICS system manager wants. Otherwise, sorry, but there are
currently no other user accessible parameters. We try to minimize the
complexity, and pitfalls, in the configuration interface. 

I am busy with other projects at the moment, but perhaps you might consider
creating a mantis entry (we want to minimize changes to R3.14 so this would
be against R3.15) describing the problem (and including a copy of this mail
exchange), and perhaps some improvements can be made (within the constraints
of reliable operation of course).

Jeff

> -----Original Message-----
> From: Mark Rivers [mailto:[email protected]]
> Sent: Friday, January 23, 2009 12:52 PM
> To: Jeff Hill; Andrew Johnson
> Cc: epics
> Subject: RE: Very slow reconnection to medm after IOC reboot
> 
> Jeff,
> 
> As I said in my first message yesterday there were 4 medm screens with a
> total of about 200 channels.
> 
> > If, after 15 minutes, the IOC comes up and the CA client library
> manages to
> > find some of its disconnected channels it has no way of knowing that
> the
> > balance of its remaining disconnected channels are on this new IOC.
> 
> It seems like the client library should know what IOC previously hosted
> the PVs that are now disconnected.  If that same IOC now comes back up,
> it seems like a reasonable assumption that the same IOC is going to be
> hosting those channels. So doing a high-rate search for those channels
> has a very high liklihood of success?
> 
> Is there an environment variable I can set to change the search rate or
> algorithm?  I agree that in general we want to avoid anything that can
> generate network storms.  But in this case I am doing development on a
> single PC, running a local client connected to channels on that same PC,
> but I have to wait 4 minutes for the channels to reconnect just because
> I rebuilt the application.  This is not acceptable.  I don't want to
> have to close and reopen all the windows in medm just so I can see what
> is going on again.
> 
> 
> Mark
> 
> -----Original Message-----
> From: Jeff Hill [mailto:[email protected]]
> Sent: Friday, January 23, 2009 12:27 PM
> To: Mark Rivers; 'Andrew Johnson'
> Cc: 'epics'
> Subject: RE: Very slow reconnection to medm after IOC reboot
> 
> 
> Mark,
> 
> How many channels were involved in this disconnect / reconnect scenario?
> 
> > Stop IOC, wait 10 seconds, restart IOC.
> > Time for medm to reconnect all channels: less than 10 seconds.
> 
> > Stop IOC, wait 15 minutes, restart IOC.
> > Time for medm to reconnect all channels: 4 minutes
> 
> When the CA library disconnects from the IOC it starts looking for the
> consequently disconnected channels with a fast search interval, but if
> the
> library isn't immediately successful it exponentially backs off to a
> much
> lower rate. We want that behavior in order to reduce the load on the
> network.
> 
> If, after 15 minutes, the IOC comes up and the CA client library manages
> to
> find some of its disconnected channels it has no way of knowing that the
> balance of its remaining disconnected channels are on this new IOC.
> Consider
> a worst case scenario; what if this particular CA client had a very
> large
> number of disconnected channels. From my perspective, it wouldn't be a
> good
> idea to search at the highest rate for these channels just because a new
> IOC
> was seen. The client library does however adopt a compromise design
> where it
> searches at an intermediate rate if it sees a beacon anomaly, and
> depending
> on how many channels are disconnected this intermediate rate might
> result in
> a 4 minute reconnect time. Furthermore, I think I recall correctly that
> the
> event of connecting to a new IOC will also cause this same intermediate
> search rate boost that is granted when a beacon anomaly event is
> detected.
> 
> Part of the rationale behind the current design is that if it was
> necessary
> to wait a long time for the IOC to come up then waiting as long as the
> number of channels divided by the number of channels per search attempt
> times this intermediate delay (maybe as much as 4 minutes in this
> situation)
> for the channels to reconnect isn't going to shatter our world. It is
> particularly important to avoid designs which cause unstable traffic
> insertion feedback when there is network congestion.
> 
> Jeff
> 
> 
> > -----Original Message-----
> > From: Mark Rivers [mailto:[email protected]]
> > Sent: Thursday, January 22, 2009 3:28 PM
> > To: Andrew Johnson
> > Cc: Jeff Hill; epics
> > Subject: RE: Very slow reconnection to medm after IOC reboot
> >
> > Hi Andrew,
> >
> > Thanks for the response.
> >
> > Here is the output of all env settings that start with the string
> EPICS,
> > before I manually set EPICS_CAS_BEACON_ADDR_LIST:
> >
> > $ printenv | grep EPICS
> > EPICS_CA_AUTO_ADDR_LIST=NO
> > EPICS_HOST_ARCH=win32-x86
> > EPICS_CA_ADDR_LIST=localhost 164.54.160.255
> > EPICS_DISPLAY_PATH=C:\EPICS\adls\
> > EPICS_CA_MAX_ARRAY_BYTES=10000000
> >
> > If I run with just those env settings I get the following message when
> I
> > boot the IOC:
> >
> > The CA server's beacon address list was empty after initialization?
> >
> > If I add the env you suggested so they now look like this:
> >
> > $ printenv | grep EPICS
> > EPICS_CA_AUTO_ADDR_LIST=NO
> > EPICS_CAS_BEACON_ADDR_LIST=164.54.160.255
> > EPICS_HOST_ARCH=win32-x86
> > EPICS_CA_ADDR_LIST=localhost 164.54.160.255
> > EPICS_DISPLAY_PATH=C:\EPICS\adls\
> > EPICS_CA_MAX_ARRAY_BYTES=10000000
> >
> > Then when I start the IOC I don't get the message about the empty
> beacon
> > address list.
> >
> > This is a new problem with 3.14.10.  I run identical IOCs built with
> > 3.14.8.2 and I don't get that error message with exactly the same
> > environment settings.
> >
> > However, even though the error message is gone the performance is only
> > marginally improved, if at all.
> >
> > Here is the result with EPICS_CAS_BEACON_ADDR_LIST set:
> >
> > ******************************
> > Windows IOC (3.14.10), Windows medm client (built with 3.14.9) on same
> > PC.
> >
> > Stop IOC, wait 10 seconds, restart IOC.
> > Time for medm to reconnect all channels: less than 10 seconds.
> >
> > Stop IOC, wait 15 minutes, restart IOC.
> > Time for medm to reconnect first channel: 10 seconds
> > Time for medm to reconnect all channels: 4 minutes
> >
> >
> > The differences from my previous test (when I had the CA server beacon
> > address error):
> >
> > The time for the very first channel to reconnect was reduced from 60
> > seconds to 10 seconds.  I am not sure how reproducible this is.
> >
> > The time for all channels to connect was reduced from 4 minutes and 25
> > seconds to 4 minutes.  Again, this may not be a statistically
> > significant improvement.
> >
> > Thus, even with EPICS_CAS_BEACON_ADDR_LIST set the performance is
> really
> > bad.  And I am not the only one seeing it, Lewis Muir reported similar
> > long reconnection times earlier this afternoon.
> >
> > Mark
> >
> >
> >
> > -----Original Message-----
> > From: Andrew Johnson [mailto:[email protected]]
> > Sent: Thursday, January 22, 2009 3:47 PM
> > To: Mark Rivers
> > Cc: Jeff Hill; epics
> > Subject: Re: Very slow reconnection to medm after IOC reboot
> >
> > On Thursday 22 January 2009 15:32:47 Mark Rivers wrote:
> > > I noticed that when I start win32-x86 IOCs built with 3.14.10 I get
> > the
> > > following message just after iocInit:
> > >
> > > *******
> > > The CA server's beacon address list was empty after initialization?
> > > *******
> > >
> > > I do not get this message when running the identical IOC built with
> > > 3.14.8.2.
> > >
> > > Is this significant?  Why am I getting this message?
> >
> > Highly, it means that the IOC is not sending out any CA beacons at
> all,
> > which
> > explains why the clients are not reconnecting to it immediately.  Try
> > setting
> > EPICS_CAS_BEACON_ADDR_LIST on the IOC to your 164.54.160.255 broadcast
> > address, although it should default to the value of EPICS_CA_ADDR_LIST
> > if not
> > set so I don't understand why it's ignoring that (unless you have
> other
> > env
> > settings for the IOC that you haven't told us about).
> >
> > - Andrew
> > --
> > The best FOSS code is written to be read by other humans -- Harold
> Welte



Replies:
Re: Very slow reconnection to medm after IOC reboot Andrew Johnson
References:
Very slow reconnection to medm after IOC reboot Mark Rivers
RE: Very slow reconnection to medm after IOC reboot Mark Rivers
Re: Very slow reconnection to medm after IOC reboot Andrew Johnson
RE: Very slow reconnection to medm after IOC reboot Mark Rivers
RE: Very slow reconnection to medm after IOC reboot Jeff Hill
RE: Very slow reconnection to medm after IOC reboot Mark Rivers

Navigate by Date:
Prev: Re: RTEMS soft reboot Till Straumann
Next: Re: RTEMS soft reboot Matt Rippa
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  <20092010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: Very slow reconnection to medm after IOC reboot J. Lewis Muir
Next: Re: Very slow reconnection to medm after IOC reboot Andrew Johnson
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  <20092010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 31 Jan 2014 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·