EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RE: CAC problem between RTEMS and vxWorks
From: "Hill, Jeff" <[email protected]>
To: Benjamin Franksen <[email protected]>, EPICS tech-talk <[email protected]>
Date: Tue, 25 Sep 2012 18:29:08 +0000
> Jeff:
> 
> Dynamic re-assign means in CA terms that the sequencer will issue a
> ca_clear_channel (for some connected channel), immediately followed by a
> ca_create_channel (to some other PV). 

Be careful about race conditions where a thread continues to use a chid 
after the channel that the chid references has been destroyed.

> Hmmm, what happens if there is a
> race between the ca_clear_channel call and the disconnect event caused by the
> IOC reboot? (Just think loud here.)
> 

Reading Wesley's later post I think we can assume that it's the client (i.e. 
sequencer) side that is being rebooted.

> It should be possible to devise a simple test client that issues a long
> sequence of such create/clear calls to check whether this is a CA problem.
> Unfortunately my automatic test setup does not yet support RTEMS, so this
> will take some time for me to do.

I had a look at the CA client side regression tests and I don't see a test
of this nature. Presumably such a test would create a few thousand channels
and then begin destroying them while at least some of them are in the process of connecting. This would be perhaps an unusual test in that we aren't testing 
any metric other than that the test program can survive a somewhat unusual 
sequence of events, and because successfully testing of anything at all might
be highly dependent on the various ratios of server/client cpu speeds and 
the speed of the network.

Considering this particular situation further. All we know is that we have 
an RTEMS IOC with a sequencer CA client that is not reconnecting reliably 
when the RTEMS IOC undergoes a rapid (close together in time) series of 
reboots. One has to guess that the vxWorks IOC is running low on some
resource perhaps its network buffers, file descriptors, ... 

Wesley:
I think that we see evidence that the vxWorks IOC is taking some time 
to determine that the previous lifetimes of the RTEMS IOC have passed out
of existence (since the RTEMS IOC didn't cleanup its CA circuits when it 
was shutdown). Have you seen the situation improve if you leave it for a 
sufficiently long interval? This would be long enough for the TCP watchdog 
timers to fire; that delay varies between different OS and different OS 
versions but is typically on the order, as I recall, of about 20 minutes.

Jeff

> 
> On Friday, September 21, 2012 11:37:03 Wesley Moore wrote:
> > From: "J. Lewis Muir" <[email protected]>
> > > There have been a number of bug fixes related to pvAssign.  You
> > > can find these in the release notes (just search in the page for
> > >
> > > "pvAssign"):
> > >   http://www-csr.bessy.de/control/SoftDist/sequencer/Notes.html
> > >
> > > What version of the sequencer are you using?
> >
> > I'm using 2-1-9.
> 
> In 2.1.9 all issues know to me have been resolved.
> 
> > Looking through all the release notes and also seq_if.c.
> > I'm thinking of trying to de-assign the channel between uses.
> 
> As a short-term work-around, this seem sensible.
> 
> Cheers
> Ben
> 
> --
> Ben Franksen
> ()  ascii ribbon campaign - against html e-mail
> /\  www.asciiribbon.org   - against proprietary attachments


Replies:
Re: CAC problem between RTEMS and vxWorks Wesley Moore
References:
Re: CAC problem between RTEMS and vxWorks Wesley Moore
Re: CAC problem between RTEMS and vxWorks Benjamin Franksen

Navigate by Date:
Prev: HP3458A Eric Norum
Next: Motor record -- URIP with soft limits Fong, Nia W.
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: CAC problem between RTEMS and vxWorks Benjamin Franksen
Next: Re: CAC problem between RTEMS and vxWorks Wesley Moore
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 18 Nov 2013 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·