EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <20112012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <20112012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RE: Sequence monitor not getting callback
From: "Shankar, Murali" <[email protected]>
To: Jeff Hill <[email protected]>, "'J. Lewis Muir'" <[email protected]>
Cc: "[email protected]" <[email protected]>
Date: Wed, 12 Oct 2011 17:15:02 -0700
On my machine, I am testing using base-3.14.12.1 and seq-2.1.2. This is indeed a multi-core CPU with 4 cores running 64 bit RedHat 6.1. Here's its /proc/version
Linux version 2.6.32-131.12.1.el6.x86_64 ([email protected]) (gcc version 4.4.5 20110214 (Red Hat 4.4.5-6) (GCC) ) #1 SMP Sun Jul 31 16:44:56 EDT 2011

I have also tested this on a 32 bit linux box running base-R3-14-12. This is also a multi-core CPU with 24 cores but running 32 bit Redhat 4. Here's its /proc/version
Linux version 2.6.9-101.ELsmp ([email protected]) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-11)) #1 SMP Fri May 27 18:57:30 EDT 2011

As far as I can tell, the CPU is not busy when I take these back traces. The CPU briefly touches 8%. As far as I can tell, there are no messages indicating that a thread exited incorrectly. 

Here's some more variations where I could reproduce the issue.
1) I have pinned this to one processor using taskset and I see the same issue. The command I am running is "taskset 01 ./bin/linux-x86_64-debug/TestnoMonitor st.cmd".
2) I changed USE_POSIX_THREAD_PRIORITY_SCHEDULING = YES and recompiled base and this test program and ran this as myself and as root.

Ernest says he'll help me run this on an single core RTEMS box and if possible, a single core VXworks box tomorrow morning. I'll post the results here. If I can find a single core linux box somewhere, I'll try this out there as well.

Regards,
Murali








-----Original Message-----
From: Jeff Hill [mailto:[email protected]] 
Sent: Wednesday, October 12, 2011 1:22 PM
To: Shankar, Murali; 'J. Lewis Muir'
Cc: [email protected]
Subject: RE: Sequence monitor not getting callback

Hi Murali,

So in both of those GDB dumps there is only one event queue processing
thread running, and both of them are parked in the normal place waiting for
a signal indicating that the queue is populated. 

That is probably a strange result if ...

o dbel is indicating at the same time that this same event queue has labor
(queue entries) pending against it. Please note also the task id diagnostic
in the dbel output and cross reference this id to what is seen in the "info
threads" output if possible.

o _and_ the stack trace for the event thread _always_, in several tries,
shows the event processing thread is in a labor pending semaphore wait
state. The event processing thread has the function "event_task" on its call
stack.

o _and_ the cpu wasn't saturated, executing in threads with higher
priorities than the event processing thread, at that time

> In both of these, the client and server ran in the same address space. In
> both of these, I launched the program, made sure we had the issue and then
> attached to the running process using "attach" from within gdb

Was the CPU busy when you captured these stack traces? Did you see any
messages indicating that a thread failed, and might have exited? Please also
send a copy of your standalone ca client test code and some instructions on
how to reproduce the problem so I can take a closer look at the situation.
Also, what version of EPICS is in use (sorry if you have already sent that
information and I lost it somehow in my mailbox). I am starting to worry
that there is an issue specific to SMP Linux. Was this running on a
multi-core CPU?

Thanks,

Jeff



Replies:
Re: Sequence monitor not getting callback J. Lewis Muir
References:
Re: Sequence monitor not getting callback Shankar, Murali
RE: Sequence monitor not getting callback Jeff Hill
RE: Sequence monitor not getting callback Shankar, Murali
Re: Sequence monitor not getting callback J. Lewis Muir
RE: Sequence monitor not getting callback Shankar, Murali
RE: Sequence monitor not getting callback Jeff Hill

Navigate by Date:
Prev: Re: Latest on archiving EPICS data Shankar, Murali
Next: Re: Latest on archiving EPICS data Paul Nord
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <20112012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: RE: Sequence monitor not getting callback Jeff Hill
Next: Re: Sequence monitor not getting callback J. Lewis Muir
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <20112012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 18 Nov 2013 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·