EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <20112012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <20112012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RE: streamdevice + asyn stuck
From: "Mark Rivers" <[email protected]>
To: "Zhang, Dehong" <[email protected]>, "Dirk Zimoch" <[email protected]>
Cc: [email protected]
Date: Tue, 12 Jul 2011 21:33:22 -0500
Hi Dehong,
 
epicsMutexShowAll 1
 
will show you if there is a mutex that is held, and the line of code where it was created.
 
How often does this happen?
 
One way to debug it is to turn on asynTrace on the underlying port driver, and send that output to a log file
 
asynSetTraceIOMask portName 0 255
asynSetTraceMask portName 0 9
asynSetTraceFile portName 0 "/home/epics/asynOutput.txt"
 
I think your asyn port name is TST:R54:GCT:16:B ?
 
The log file may get quite large, but at the end you should see what goes wrong.
 
Mark
 

 
________________________________

From: Zhang, Dehong [mailto:[email protected]]
Sent: Tue 7/12/2011 8:36 PM
To: Dirk Zimoch
Cc: Mark Rivers; [email protected]
Subject: RE: streamdevice + asyn stuck



Hi Dirk and Mark,

Thank you very much for helping!

I tried to change the SCAN to "10 second", it did not help.  I also tried the
epicsThreadShowAll command as Dirk suggested, it shows a few timerQueue
threads, all "OK"

>From what I can see from the source code, I believe that we somehow
lose callbacks from asyn, that breaks the evalcommand() chain and we
never get to finishProtocol().

Another possibility is the mutex.  If it is not released properly, the
Stream::process() method will be stuck waiting.

Best regards,
Dehong




________________________________________
From: Dirk Zimoch [[email protected]]
Sent: Tuesday, July 12, 2011 12:21 AM
To: Zhang, Dehong
Cc: [email protected]; [email protected]
Subject: Re: streamdevice + asyn stuck

Hello Dehong,

Do you have a hung up timerQueue thread? Try epicsThreadShowAll.

Dirk

Zhang, Dehong wrote:
> Hi Dirk, Mark and fellow EPICSers,
>
> Sorry to bother you with this.  I tried to chase the source code and search the tech-talk,
> but could not find much hints.
>
> We use base 3.14.9, streamdevice 2.4 and asyn 4.10, have been experiencing this stuck
> problem for some time.  The symptom is that randomly, a mbbi/ai/bi... (input) record would
> stop updating and no longer does FLNK, with no error/warning messages printed in the log
> file.  And dbpr *** 3 would show that:
> LCNT=11
> SEVR=INVALID
> STAT=SCAN
>
> And manually writing to the PROC field also does nothing.
>
> It seems like the record is stuck waiting for a callback from streamdevice (and asyn), and
> the EPICS framework just ignores any "PROC" requests.
>
> In the protocol file we do have the timeouts etc:
> locktimeout  = 5000;
> terminator   = CR;
> replytimeout = 1000;
> readtimeout  = 1000;
> extrainput   = Ignore;
>
> While chasing the problem, I noticed that "asynReport 10 ***" would show:
> multiDevice:No canBlock:Yes autoConnect:Yes
>     enabled:Yes connected:Yes numberConnects 1
>
> For some the numberConnects can be 2.
>
> Is this "canBlock:Yes" related to my problem?  Should the numberConnects be <=1?
> We use one port to talk to one device.
>
> Rebooting the IOC does fix the problems, but then they would come back, randomly.
>
> Please advise.  Thank you very much.
>
> Best regards,
> Dehong
>





References:
streamdevice + asyn stuck Zhang, Dehong
Re: streamdevice + asyn stuck Dirk Zimoch
RE: streamdevice + asyn stuck Zhang, Dehong

Navigate by Date:
Prev: RE: streamdevice + asyn stuck Zhang, Dehong
Next: CSS errors: cannot excute Ma Xiaoyuan
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <20112012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: RE: streamdevice + asyn stuck Zhang, Dehong
Next: Re: streamdevice + asyn stuck Dirk Zimoch
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <20112012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 18 Nov 2013 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·