Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  <20102011  2012  2013  2014  2015  2016  2017  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  <20102011  2012  2013  2014  2015  2016  2017 
<== Date ==> <== Thread ==>

Subject: Re: No monitor refresh
From: Hinko Kocevar <hinko.kocevar@i-tech.si>
To: Mark Rivers <rivers@cars.uchicago.edu>
Cc: tech-talk@aps.anl.gov
Date: Thu, 13 May 2010 12:16:15 +0200
Hello Mark,


On 05/12/10 13:29, Mark Rivers wrote:
>> Since the I/O Intr scans are used could it be that they do not get sent
>> out due to high CPU load? For example by changing the scanning to 1
>> second seems to remedy the situation - scanning periodically is more
>> deterministic and possibly with higher priority?
>  
> The callbacks to device support are done by your driver.  My guess is that those are happening OK.  But those callbacks result in a call to scanIoRequest, asking EPICS to process the record.  That processing is done in one of the EPICS callbacks tasks (cbLow?).  If the CPU is very busy then it is possible that the callback task never gets to run, and the record never gets processed.  

Actual execution of the PV processing must be 'lost' somewhere along the
way, because EPICS receives the event from lower layer, but does not
react to it - process the PV!?

>  
>> Asyn version is 4-9.
> 
> Aha!  This is from the asyn R4-10 release notes:
> 
> 
> **************************
> 
> 
> devEpics
> 
> 
> Undid the change that was done in R4-9 with direct calls to dbScanLock and process in the interrupt callback functions. This could lead to deadlocks in some circumstances. The original reason for changing from scanIoRequest to (dbScanLock, process) was because callback values would be lost if the callbacks came so close together in time that the single callback value stored in device support was overwritten before scanIoRequest could process the record. This problem has been fixed by adding a FIFO (ring buffer) to the device support for the scalar interfaces asynInt32, asynUInt32Digital, and asynFloat64. The ring buffer is only created when the record is put into I/O Intr scan, so the storage is not allocated for records that are not I/O Intr scanned. The ring buffer default size is 10 values, but this can be changed on a per-record basis using the dbInfo string "FIFO" with a value such as "100".
> 
> 
> **************************
> 
> So your problem could be a known bug which was fixed in 4-10, September 2008.

Thank your for this info!
I'll rebuild the EPICS + Asyn with newer version and see how it behaves
then..

> 
>> I'm not that familiar with EPICS internals - is the interruptAccept some
>> global variable I can rely on? Can you maybe post some code on how your
>> driver handles the interrupt?
> 
> Yes, interruptAccept is such a global variable.  The code that handles the interrupts is included in newer versions of asyn, in asyn/miscellaneous/asynPortDriver.cpp.

Fine then - I'll try to rely on the interruptAccept before starting to
handle interrupts..




Thank you!

Best regards,
Hinko



> 
> 
> Mark
>  
> 
> 
> ________________________________
> 
> From: Hinko Kocevar [mailto:hinko.kocevar@i-tech.si]
> Sent: Wed 5/12/2010 1:22 AM
> To: Mark Rivers
> Cc: Ralph Lange; tech-talk@aps.anl.gov
> Subject: Re: No monitor refresh
> 
> 
> 
> Dear Mark,
> 
> On 05/11/10 23:24, Mark Rivers wrote:
>>>> When you do intermediate cagets while your camonitor is not getting
>>>> updates, what timestamps are these showing?
>>
>>> They seem to be showing some old timestamp - e.g. few hours back,
>> while
>>> it should show current date&time stamp.
>>
>> That really suggests that your record is not processing, i.e. that the
>> problem is not with Channel Access callbacks to clients, but with
>> callbacks from your driver to device support.
> 
> Right.
> 
> Since the I/O Intr scans are used could it be that they do not get sent
> out due to high CPU load? For example by changing the scanning to 1
> second seems to remedy the situation - scanning periodically is more
> deterministic and possibly with higher priority?
> 
>>
>>>> Are you sure asyn gets the value you expect it to into the record?
>>>> (Check by enabling asyn debug.)
>>
>>> I've checked the status of the low level stuff (kernel driver and
>>> hardware) and the new value is there..
>>
>> But you didn't answer the question: does the RECORD have the correct
>> value?  Do a "dbpr" on the record at the IOC prompt.  If the record does
>> not have the correct value then the problem is not with Channel Access
>> callbacks, it is with asyn callbacks from your driver to the record.
> 
> No, the monitored record DOES NOT have the correct value - hence the old
> time stamp with the old value, I reckon.
> 
>>
>> You should probably put some printf calls in your driver where it does
>> the interrupt callbacks, and/or in devAsynInt32.c where it gets
>> interrupt callbacks from the driver.
> 
> Will do.
> 
> FYI, I followed Jeff's advice and checked the CPU load - one of the
> units has constant load over 90% and this is the offending one, that
> does not send out CA_PROTO_EVENT_ADD responses - probably because the
> monitor did not see the interrupt callback yet!?.
> 
> By relaxing the EPICS IOC work, CPU load dropped below 50% and the
> CA_PROTO_EVENT_ADD responses start arriving smoothly!
> 
> I'll need to investigate further to be sure what the cause/remedy will be.
> 
> 
> 
> 
> Best regards,
> Hinko
> 
> 
> 
>>
>> Mark
>>
>> -----Original Message-----
>> From: tech-talk-bounces@aps.anl.gov
>> [mailto:tech-talk-bounces@aps.anl.gov] On Behalf Of Hinko Kocevar
>> Sent: Tuesday, May 11, 2010 11:01 AM
>> To: Ralph Lange
>> Cc: tech-talk@aps.anl.gov
>> Subject: Re: No monitor refresh
>>
>> Hello Ralph,
>>
>> On 05/11/10 16:47, Ralph Lange wrote:
>>> Hello Hinko,
>>>
>>> hm ..... it's an I/O scanned mbbi connected to asyn, and I have no
>> asyn
>>> experience. (So I'm sort of out-of-the-game at this point.)
>>
>> Yes it is a record defined like this:
>>
>> record(mbbi, "$(DEVICE)ENV_AGC_MONITOR") {
>>       field(DTYP, "asynInt32")
>>       field(INP, "@asyn($(PORT), 725, 1.0)")
>>       field(SCAN, "I/O Intr")
>>       field(DESC, "AGC")
>>       field(ZRVL, "0")
>>       field(ONVL, "1")
>>       field(ZRST, "MANUAL")
>>       field(ONST, "AUTO")
>>       field(PINI, "YES")
>> }
>>
>> record(mbbo, "$(DEVICE)ENV_AGC_SP") {
>>       field(DTYP, "asynInt32")
>>       field(OUT, "@asyn($(PORT), 725)")
>>       field(DESC, "AGC")
>>       field(ZRVL, "0")
>>       field(ONVL, "1")
>>       field(ZRST, "MANUAL")
>>       field(ONST, "AUTO")
>>       field(PINI, "YES")
>> }
>>
>> ENV_AGC_MONITOR is used with camonitor, ENV_AGC_SP is used with
>> caget/caput. When I set the value with ENV_AGC_SP, ENV_AGC_MONITOR
>> should refresh the value.
>>
>>>
>>> Nevertheless - some thoughts:
>>> Your NOBT is zero, but your mask is correct, probably something that
>>> asyn does for you.
>>> Looked at the record code: An mbbi should definitely send VAL updates
>>> when it processes and the value changed.
>>
>> Yes it should, but this does not happen always.
>>
>>> Are you sure asyn gets the value you expect it to into the record?
>>> (Check by enabling asyn debug.)
>>
>> I've checked the status of the low level stuff (kernel driver and
>> hardware) and the new value is there..
>>
>>> Are you sure the record gets processed by asyn when you expect it to?
>>> (Check by setting the trace field.)
>>
>> I want it to react as soon as the value in hardware changes - hence the
>> I/O scanning. Hmm, maybe I could try some other Periodic scans to see if
>> it helps..
>>
>> Will try the asyn debug and traces too..
>>
>>> When you set up multiple camonitor clients - do they all behave the
>> same?
>>
>> Yes, they do.
>>
>> I've also tried running camonitor on several clients, even locally on
>> the IOC server itself (ARM cross compiled), but without any success. It
>> seems to me that EPICS TCP transport layer is not reacting to the change
>> !?
>>
>>> When you do intermediate cagets while your camonitor is not getting
>>> updates, what timestamps are these showing?
>>
>> They seem to be showing some old timestamp - e.g. few hours back, while
>> it should show current date&time stamp.
>>
>>> (BTW: You only have two states with raw values 0 and 1 defined - any
>>> particular reason for using mbbi as opposed to bi?)
>>
>> No particular reason at all.. :)
>>
>>>
>>> Someone with more asyn background could probably say more....
>>
>>
>> Thank you for the valuable pointers!
>>
>> Best regards,
>> Hinko
>>
>>
>>>
>>> Good luck!
>>> Ralph
>>>
>>>
>>> On Tue 11 May 2010 10:07:12 Hinko Kocevar wrote:
>>>> Hello Ralph,
>>>>
>>>> On 05/11/10 15:21, Ralph Lange wrote:
>>>>
>>>>> Hello Hinko,
>>>>>
>>>>> what EPICS versions are you using for client and IOC?
>>>>>    
>>>>
>>>> IOC and client use 3.14.10 EPICS base version.
>>>>
>>>>
>>>>> Can you show a dump (dbpr <name>, 4) of the record that you are
>>>>> connecting to?
>>>>>    
>>>>
>>>> Sure, here it is:
>>>> epics> dbpr LIBERA01:ENV:ENV_AGC_MONITOR 4
>>>> ACKS: NO_ALARM      ACKT: YES           ASG:                ASP:
>>>> 0x00000000
>>>> BKPT: 0x00          COSV: NO_ALARM      DESC: AGC           DISA: 0
>>>>
>>>> DISP: 0             DISS: NO_ALARM      DISV: 1             DPVT:
>>>> 0x50c72c00
>>>> DSET: 0x7c620840    DTYP: asynInt32     EIST:               EISV:
>>>> NO_ALARM
>>>> EIVL: 0x0           ELST:               ELSV: NO_ALARM      ELVL: 0x0
>>>>
>>>> EVNT: 0             FFST:               FFSV: NO_ALARM      FFVL: 0x0
>>>>
>>>> FLNK:CONSTANT 0     FRST:               FRSV: NO_ALARM      FRVL: 0x0
>>>>
>>>> FTST:               FTSV: NO_ALARM      FTVL: 0x0           FVST:
>>>>
>>>> FVSV: NO_ALARM      FVVL: 0x0           INP:INST_IO @asyn(ENV, 725,
>> 1.0)
>>>>
>>>> LALM: 1             LCNT: 0             LSET: 0xe8752e00    MASK:
>>>> 4294967295
>>>> MLIS: 0x38d0320060d0320003000000        MLOK: 0x28700b00    MLST: 1
>>>>
>>>> NAME: LIBERA01:ENV:ENV_AGC_MONITOR      NIST:               NISV:
>>>> NO_ALARM
>>>> NIVL: 0x0           NOBT: 0             NSEV: NO_ALARM      NSTA:
>>>> NO_ALARM
>>>> ONST: AUTO          ONSV: NO_ALARM      ONVL: 0x1           ORAW: 1
>>>>
>>>> PACT: 0             PHAS: 0             PINI: YES           PPN:
>>>> 0x00000000
>>>> PPNR: 0x00000000    PRIO: LOW           PROC: 0             PUTF: 0
>>>>
>>>> RDES: 0xa8a10300    RPRO: 0             RSET: 0x34b70a40    RVAL: 1
>>>>
>>>> SCAN: I/O Intr      SDEF: 1             SDIS:CONSTANT       SEVR:
>>>> NO_ALARM
>>>> SHFT: 0             SIML:CONSTANT       SIMM: NO            SIMS:
>>>> NO_ALARM
>>>> SIOL:CONSTANT       SPVT: 0x388a2f00    STAT: NO_ALARM      SVAL: 0
>>>>
>>>> SVST:               SVSV: NO_ALARM      SVVL: 0x0           SXST:
>>>>
>>>> SXSV: NO_ALARM      SXVL: 0x0           TEST:               TESV:
>>>> NO_ALARM
>>>> TEVL: 0x0           THST:               THSV: NO_ALARM      THVL: 0x0
>>>>
>>>> TIME: 2010-05-11 11:18:40.100440000     TPRO: 0             TSE: 0
>>>>
>>>> TSEL:CONSTANT       TTST:               TTSV: NO_ALARM      TTVL: 0x0
>>>>
>>>> TVST:               TVSV: NO_ALARM      TVVL: 0x0           TWST:
>>>>
>>>> TWSV: NO_ALARM      TWVL: 0x0           UDF: 0              UNSV:
>>>> NO_ALARM
>>>> VAL: 1              ZRST: MANUAL        ZRSV: NO_ALARM      ZRVL: 0x0
>>>>
>>>>
>>>>
>>>>> What field of that record are you connecting to?
>>>>>    
>>>>
>>>> I did no specify any field - so it is VAL, I guess by default.
>>>>
>>>> Thank you!
>>>>
>>>> Regards,
>>>> Hinko
>>>>
>>>>
>>>>> Regards,
>>>>> Ralph
>>>>>
>>>>>
>>>>> On Tue 11 May 2010 6:26:48 Hinko Kocevar wrote:
>>>>>   
>>>>>> Hi all,
>>>>>>
>>>>>> I'm experiencing some strange behavior when using camonitor against
>> an
>>>>>> EPICS IOC.
>>>>>>
>>>>>> For some reason a CA_PROTO_EVENT_ADD response from IOC to CA client
>>>>>> does
>>>>>> not arrive, or arrives several minutes later after the PV value has
>>>>>> changed. Note that first CA_PROTO_EVENT_ADD response is seen
>> always,
>>>>>> only latter CA_PROTO_EVENT_ADD responses do not arrive (or arrive
>>>>>> late)?!
>>>>>> This behavior is not seen always - sometimes the IOC would promptly
>>>>>> respond with CA_PROTO_EVENT_ADD response when needed.
>>>>>>
>>>>>> I can see the CA_PROTO_ECHO requests/responses fine on the same
>>>>>> camonitor channel link, only CA_PROTO_EVENT_ADD response is not
>> seen.
>>>>>>
>>>>>> Inspecting the IOC I can see that the value did change (on lower
>>>>>> levels)
>>>>>> and caput response is also valid - updated with new value.
>>>>>>
>>>>>> I've ruled out the network problems, since the issue is raised even
>>>>>> when
>>>>>> using cross-over cable to communicate with the IOC. Actually even I
>>>>>> update the value in the IOC shell, the camonitor is not refreshed!!
>>>>>>
>>>>>>
>>>>>>
>>>>>> Has someone experienced something similar?
>>>>>> Are there some conditions that need to fulfilled in order for IOC
>> to
>>>>>> send a subsequent CA_PROTO_EVENT_ADD response(s)?
>>>>>>
>>>>>>
>>>>>>
>>>>>> TIA & HAND,
>>>>>> Hinko
>>>>>>      
>>>>
>>>>
>>>>  
>>
>>
> 
> 
> --
> Hinko Kocevar
> Technical support software engineer
> Instrumentation Technologies
> Velika pot 22, SI-5250 Solkan - Slovenia
> T:+386 5 3352600, F:+386 5 3352601
> mailto: hinko.kocevar@i-tech.si
> 
> http://www.i-tech.si <http://www.i-tech.si/>  - When your users demand stability
> 
> The information transmitted is intended solely for the addressee and may
> contain confidential and/or privileged information. Any review, retention,
> disclosure or other use by persons other than the intended recipient is
> prohibited. If you received this in error, please notify the sender and
> delete all copies.
> 
> 


-- 
Hinko Kocevar
Technical support software engineer
Instrumentation Technologies
Velika pot 22, SI-5250 Solkan - Slovenia
T:+386 5 3352600, F:+386 5 3352601
mailto: hinko.kocevar@i-tech.si

http://www.i-tech.si - When your users demand stability

The information transmitted is intended solely for the addressee and may
contain confidential and/or privileged information. Any review, retention,
disclosure or other use by persons other than the intended recipient is
prohibited. If you received this in error, please notify the sender and
delete all copies.

Replies:
Re: No monitor refresh Hinko Kocevar
References:
Re: No monitor refresh Ralph Lange
Re: No monitor refresh Hinko Kocevar
RE: No monitor refresh Mark Rivers
Re: No monitor refresh Hinko Kocevar
RE: No monitor refresh Mark Rivers

Navigate by Date:
Prev: Re: Gateway compilation issue in tsHash.h Carl Schumann
Next: *** buffer overflow detected ***: ./ioc terminated tanushyam bhattacharjee
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  <20102011  2012  2013  2014  2015  2016  2017 
Navigate by Thread:
Prev: RE: No monitor refresh Mark Rivers
Next: Re: No monitor refresh Hinko Kocevar
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  <20102011  2012  2013  2014  2015  2016  2017 
ANJ, 02 Sep 2010 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·