Experimental Physics and
Industrial Control System

"Mark Rivers" <[email protected]> · Fri, 23 Jan 2009 15:02:16 -0600

Folks,

Andrew pointed out to me today that recent versions of medm have a
"Retry connections" button that one can access by right-clicking on a
running display.  That sounded like a good workaround for getting medm
displays back up.

I just tried that.  2 minutes after rebooting an IOC, some widgets on
the medm window are still disconnected, while some are connected.
Performing "Retry connections" did not help, the channels remain
disconnected.  This is medm 3.1.2.1 built with 3.14.9 on Windows.

Jeff, do you understand why when I run a Windows IOC with
EPICS_CA_AUTO_ADDR_LIST=NO
EPICS_CA_ADDR_LIST=localhost 164.54.160.255

I get the following message on boot:

The CA server's beacon address list was empty after initialization?

This only happens on 3.14.10, not 3.14.8.2.  Same computer, same cmd
shell window, just switch between an old application and a new one.

When that error happens is the CA library actually sending any beacons?

On Andrew's suggestion I added
EPICS_CAS_BEACON_ADDR_LIST=164.54.160.255

That eliminates the error message, but results in no measureable
improvement in reconnection time (still 4 minutes).  Is this expected?

Thanks,
Mark

-----Original Message-----
From: Mark Rivers 
Sent: Friday, January 23, 2009 1:52 PM
To: 'Jeff Hill'; 'Andrew Johnson'
Cc: 'epics'
Subject: RE: Very slow reconnection to medm after IOC reboot

Jeff,

As I said in my first message yesterday there were 4 medm screens with a
total of about 200 channels.

> If, after 15 minutes, the IOC comes up and the CA client library
manages to
> find some of its disconnected channels it has no way of knowing that
the
> balance of its remaining disconnected channels are on this new IOC.

It seems like the client library should know what IOC previously hosted
the PVs that are now disconnected.  If that same IOC now comes back up,
it seems like a reasonable assumption that the same IOC is going to be
hosting those channels. So doing a high-rate search for those channels
has a very high liklihood of success?

Is there an environment variable I can set to change the search rate or
algorithm?  I agree that in general we want to avoid anything that can
generate network storms.  But in this case I am doing development on a
single PC, running a local client connected to channels on that same PC,
but I have to wait 4 minutes for the channels to reconnect just because
I rebuilt the application.  This is not acceptable.  I don't want to
have to close and reopen all the windows in medm just so I can see what
is going on again.

Mark

-----Original Message-----
From: Jeff Hill [mailto:[email protected]] 
Sent: Friday, January 23, 2009 12:27 PM
To: Mark Rivers; 'Andrew Johnson'
Cc: 'epics'
Subject: RE: Very slow reconnection to medm after IOC reboot

Mark,

How many channels were involved in this disconnect / reconnect scenario?

> Stop IOC, wait 10 seconds, restart IOC.
> Time for medm to reconnect all channels: less than 10 seconds.

> Stop IOC, wait 15 minutes, restart IOC.
> Time for medm to reconnect all channels: 4 minutes

When the CA library disconnects from the IOC it starts looking for the
consequently disconnected channels with a fast search interval, but if
the
library isn't immediately successful it exponentially backs off to a
much
lower rate. We want that behavior in order to reduce the load on the
network.

If, after 15 minutes, the IOC comes up and the CA client library manages
to
find some of its disconnected channels it has no way of knowing that the
balance of its remaining disconnected channels are on this new IOC.
Consider
a worst case scenario; what if this particular CA client had a very
large
number of disconnected channels. From my perspective, it wouldn't be a
good
idea to search at the highest rate for these channels just because a new
IOC
was seen. The client library does however adopt a compromise design
where it
searches at an intermediate rate if it sees a beacon anomaly, and
depending
on how many channels are disconnected this intermediate rate might
result in
a 4 minute reconnect time. Furthermore, I think I recall correctly that
the
event of connecting to a new IOC will also cause this same intermediate
search rate boost that is granted when a beacon anomaly event is
detected.

Part of the rationale behind the current design is that if it was
necessary
to wait a long time for the IOC to come up then waiting as long as the
number of channels divided by the number of channels per search attempt
times this intermediate delay (maybe as much as 4 minutes in this
situation)
for the channels to reconnect isn't going to shatter our world. It is
particularly important to avoid designs which cause unstable traffic
insertion feedback when there is network congestion.

Jeff

> -----Original Message-----
> From: Mark Rivers [mailto:[email protected]]
> Sent: Thursday, January 22, 2009 3:28 PM
> To: Andrew Johnson
> Cc: Jeff Hill; epics
> Subject: RE: Very slow reconnection to medm after IOC reboot
> 
> Hi Andrew,
> 
> Thanks for the response.
> 
> Here is the output of all env settings that start with the string
EPICS,
> before I manually set EPICS_CAS_BEACON_ADDR_LIST:
> 
> $ printenv | grep EPICS
> EPICS_CA_AUTO_ADDR_LIST=NO
> EPICS_HOST_ARCH=win32-x86
> EPICS_CA_ADDR_LIST=localhost 164.54.160.255
> EPICS_DISPLAY_PATH=C:\EPICS\adls\
> EPICS_CA_MAX_ARRAY_BYTES=10000000
> 
> If I run with just those env settings I get the following message when
I
> boot the IOC:
> 
> The CA server's beacon address list was empty after initialization?
> 
> If I add the env you suggested so they now look like this:
> 
> $ printenv | grep EPICS
> EPICS_CA_AUTO_ADDR_LIST=NO
> EPICS_CAS_BEACON_ADDR_LIST=164.54.160.255
> EPICS_HOST_ARCH=win32-x86
> EPICS_CA_ADDR_LIST=localhost 164.54.160.255
> EPICS_DISPLAY_PATH=C:\EPICS\adls\
> EPICS_CA_MAX_ARRAY_BYTES=10000000
> 
> Then when I start the IOC I don't get the message about the empty
beacon
> address list.
> 
> This is a new problem with 3.14.10.  I run identical IOCs built with
> 3.14.8.2 and I don't get that error message with exactly the same
> environment settings.
> 
> However, even though the error message is gone the performance is only
> marginally improved, if at all.
> 
> Here is the result with EPICS_CAS_BEACON_ADDR_LIST set:
> 
> ******************************
> Windows IOC (3.14.10), Windows medm client (built with 3.14.9) on same
> PC.
> 
> Stop IOC, wait 10 seconds, restart IOC.
> Time for medm to reconnect all channels: less than 10 seconds.
> 
> Stop IOC, wait 15 minutes, restart IOC.
> Time for medm to reconnect first channel: 10 seconds
> Time for medm to reconnect all channels: 4 minutes
> 
> 
> The differences from my previous test (when I had the CA server beacon
> address error):
> 
> The time for the very first channel to reconnect was reduced from 60
> seconds to 10 seconds.  I am not sure how reproducible this is.
> 
> The time for all channels to connect was reduced from 4 minutes and 25
> seconds to 4 minutes.  Again, this may not be a statistically
> significant improvement.
> 
> Thus, even with EPICS_CAS_BEACON_ADDR_LIST set the performance is
really
> bad.  And I am not the only one seeing it, Lewis Muir reported similar
> long reconnection times earlier this afternoon.
> 
> Mark
> 
> 
> 
> -----Original Message-----
> From: Andrew Johnson [mailto:[email protected]]
> Sent: Thursday, January 22, 2009 3:47 PM
> To: Mark Rivers
> Cc: Jeff Hill; epics
> Subject: Re: Very slow reconnection to medm after IOC reboot
> 
> On Thursday 22 January 2009 15:32:47 Mark Rivers wrote:
> > I noticed that when I start win32-x86 IOCs built with 3.14.10 I get
> the
> > following message just after iocInit:
> >
> > *******
> > The CA server's beacon address list was empty after initialization?
> > *******
> >
> > I do not get this message when running the identical IOC built with
> > 3.14.8.2.
> >
> > Is this significant?  Why am I getting this message?
> 
> Highly, it means that the IOC is not sending out any CA beacons at
all,
> which
> explains why the clients are not reconnecting to it immediately.  Try
> setting
> EPICS_CAS_BEACON_ADDR_LIST on the IOC to your 164.54.160.255 broadcast
> address, although it should default to the value of EPICS_CA_ADDR_LIST
> if not
> set so I don't understand why it's ignoring that (unless you have
other
> env
> settings for the IOC that you haven't told us about).
> 
> - Andrew
> --
> The best FOSS code is written to be read by other humans -- Harold
Welte

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System