Experimental Physics and
Industrial Control System

"Shankar, Murali" <[email protected]> · Wed, 19 Oct 2011 16:38:05 -0700

Thank you, Jeff!

I can confirm that this fixes the issue. I downloaded the latest from lp:epics-base/3.14 and rebuilt my tests. Both the CA only test and the more complex test with the sequencer work well. 

I also generated a patch with Jeff's changes and applied it to a snapshot of epics-base (patched base-R3-14-12) that we use in production and tested both the CA only test and the more complex test with the sequencer. In 10 tries of the test with the sequencer, I did not see the problem even once. 

Ernest and I will push this thru to production; we'll then make sure this fixes Mike Z's original issue with his sIOC's. 

Many thanks to Jeff for the quick turnaround and to Ben/others for their assistance in debugging this issue. 

Regards,
Murali

-----Original Message-----
From: Jeff Hill [mailto:[email protected]] 
Sent: Wednesday, October 19, 2011 11:30 AM
To: Shankar, Murali; 'J. Lewis Muir'
Cc: [email protected]
Subject: RE: Sequence monitor not getting callback

All,

Thanks to Murali Shankar a serious bug has been detected in EPICS base. 

Recent changes in the sequencer has caused it to subscribe to in-memory
(in-IOC) channels from multiple threads and subsequently it has been
discovered at SLAC that this wasn't working reliable with some of the
subscriptions not receiving their first, or any, updates when there are
large numbers of channels in the sequencer. The bug is probably present in
all currently released versions of EPICS base, but possibly only encountered
with recent releases of the sequencer.

I have committed a fix to the event queue components of the R3.14 branch of
EPICS base for this problem, and also for another minor issue detected when
running my regression tests against in-memory (in-IOC) channels. I also
committed a change adding a new test for Murali's issue to my regression
tests, and also a new adapter code which optionally registers the ca client
side regression tests with the IOC shell. 

Many thanks again to Murali for her diligence in preparing a test code which
reproduces the issue.

Ref https://bugs.launchpad.net/epics-base/+bug/878372
Ref https://bugs.launchpad.net/epics-base/+bug/878387

Jeff
______________________________________________________
Jeffrey O. Hill           Email        [email protected]
LANL MS H820              Voice        505 665 1831
Los Alamos NM 87545 USA   FAX          505 665 5107

Message content: Correspondence

> -----Original Message-----
> From: Shankar, Murali [mailto:[email protected]]
> Sent: Thursday, October 13, 2011 12:28 PM
> To: J. Lewis Muir
> Cc: Jeff Hill; [email protected]
> Subject: RE: Sequence monitor not getting callback
> 
> Thank you. I tried this out on my dev box and /proc/cpuinfo happily
> reports only one CPU.
> 
> The test with CA alone performs better in that it succeeds about 50% of
> the time. But a little less than 50% of the time, I still get monitors
> that have not received callbacks.
> 
> The original test with sequence programs performs much better. So far, I
> have been able to reproduce the issue only once in about 10 tests. But the
> issue still occurs.
> 
> I have attached the thread dumps as well.
> 
> Regards,
> Murali
> 
> 
> 
> 
> -----Original Message-----
> From: J. Lewis Muir [mailto:[email protected]]
> Sent: Thursday, October 13, 2011 7:52 AM
> To: Shankar, Murali
> Cc: Jeff Hill; [email protected]
> Subject: Re: Sequence monitor not getting callback
> 
> On 10/12/11 7:15 PM, Shankar, Murali wrote:
> > Ernest says he'll help me run this on an single core RTEMS box
> > and if possible, a single core VXworks box tomorrow morning.
> > I'll post the results here. If I can find a single core linux
> > box somewhere, I'll try this out there as well.
> 
> Hi, Murali.
> 
> I haven't tried it, but I think many SMP Linux kernels support
> the 'maxcpus' boot parameter (or the older 'nosmp' boot
> parameter).  So perhaps you could use this option to effectively
> change your Linux box into a single CPU box for your test?
> 
> See
> 
> http://www.kernel.org/doc/Documentation/kernel-parameters.txt
> 
> and the bootparam(7) man page.
> 
> Lewis

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System