Experimental Physics and
Industrial Control System

"Mark Rivers" <[email protected]> · Wed, 12 May 2010 06:29:26 -0500

> Since the I/O Intr scans are used could it be that they do not get sent
> out due to high CPU load? For example by changing the scanning to 1
> second seems to remedy the situation - scanning periodically is more
> deterministic and possibly with higher priority?

The callbacks to device support are done by your driver.  My guess is that those are happening OK.  But those callbacks result in a call to scanIoRequest, asking EPICS to process the record.  That processing is done in one of the EPICS callbacks tasks (cbLow?).  If the CPU is very busy then it is possible that the callback task never gets to run, and the record never gets processed.  

> Asyn version is 4-9.

Aha!  This is from the asyn R4-10 release notes:

**************************

devEpics

Undid the change that was done in R4-9 with direct calls to dbScanLock and process in the interrupt callback functions. This could lead to deadlocks in some circumstances. The original reason for changing from scanIoRequest to (dbScanLock, process) was because callback values would be lost if the callbacks came so close together in time that the single callback value stored in device support was overwritten before scanIoRequest could process the record. This problem has been fixed by adding a FIFO (ring buffer) to the device support for the scalar interfaces asynInt32, asynUInt32Digital, and asynFloat64. The ring buffer is only created when the record is put into I/O Intr scan, so the storage is not allocated for records that are not I/O Intr scanned. The ring buffer default size is 10 values, but this can be changed on a per-record basis using the dbInfo string "FIFO" with a value such as "100".

**************************

So your problem could be a known bug which was fixed in 4-10, September 2008.

> I'm not that familiar with EPICS internals - is the interruptAccept some
> global variable I can rely on? Can you maybe post some code on how your
> driver handles the interrupt?

Yes, interruptAccept is such a global variable.  The code that handles the interrupts is included in newer versions of asyn, in asyn/miscellaneous/asynPortDriver.cpp.

Mark

________________________________

From: Hinko Kocevar [mailto:[email protected]]
Sent: Wed 5/12/2010 1:22 AM
To: Mark Rivers
Cc: Ralph Lange; [email protected]
Subject: Re: No monitor refresh

Dear Mark,

On 05/11/10 23:24, Mark Rivers wrote:
>>> When you do intermediate cagets while your camonitor is not getting
>>> updates, what timestamps are these showing?
>
>> They seem to be showing some old timestamp - e.g. few hours back,
> while
>> it should show current date&time stamp.
>
> That really suggests that your record is not processing, i.e. that the
> problem is not with Channel Access callbacks to clients, but with
> callbacks from your driver to device support.

Right.

Since the I/O Intr scans are used could it be that they do not get sent
out due to high CPU load? For example by changing the scanning to 1
second seems to remedy the situation - scanning periodically is more
deterministic and possibly with higher priority?

>
>>> Are you sure asyn gets the value you expect it to into the record?
>>> (Check by enabling asyn debug.)
>
>> I've checked the status of the low level stuff (kernel driver and
>> hardware) and the new value is there..
>
> But you didn't answer the question: does the RECORD have the correct
> value?  Do a "dbpr" on the record at the IOC prompt.  If the record does
> not have the correct value then the problem is not with Channel Access
> callbacks, it is with asyn callbacks from your driver to the record.

No, the monitored record DOES NOT have the correct value - hence the old
time stamp with the old value, I reckon.

>
> You should probably put some printf calls in your driver where it does
> the interrupt callbacks, and/or in devAsynInt32.c where it gets
> interrupt callbacks from the driver.

Will do.

FYI, I followed Jeff's advice and checked the CPU load - one of the
units has constant load over 90% and this is the offending one, that
does not send out CA_PROTO_EVENT_ADD responses - probably because the
monitor did not see the interrupt callback yet!?.

By relaxing the EPICS IOC work, CPU load dropped below 50% and the
CA_PROTO_EVENT_ADD responses start arriving smoothly!

I'll need to investigate further to be sure what the cause/remedy will be.

Best regards,
Hinko

>
> Mark
>
> -----Original Message-----
> From: [email protected]
> [mailto:[email protected]] On Behalf Of Hinko Kocevar
> Sent: Tuesday, May 11, 2010 11:01 AM
> To: Ralph Lange
> Cc: [email protected]
> Subject: Re: No monitor refresh
>
> Hello Ralph,
>
> On 05/11/10 16:47, Ralph Lange wrote:
>> Hello Hinko,
>>
>> hm ..... it's an I/O scanned mbbi connected to asyn, and I have no
> asyn
>> experience. (So I'm sort of out-of-the-game at this point.)
>
> Yes it is a record defined like this:
>
> record(mbbi, "$(DEVICE)ENV_AGC_MONITOR") {
>       field(DTYP, "asynInt32")
>       field(INP, "@asyn($(PORT), 725, 1.0)")
>       field(SCAN, "I/O Intr")
>       field(DESC, "AGC")
>       field(ZRVL, "0")
>       field(ONVL, "1")
>       field(ZRST, "MANUAL")
>       field(ONST, "AUTO")
>       field(PINI, "YES")
> }
>
> record(mbbo, "$(DEVICE)ENV_AGC_SP") {
>       field(DTYP, "asynInt32")
>       field(OUT, "@asyn($(PORT), 725)")
>       field(DESC, "AGC")
>       field(ZRVL, "0")
>       field(ONVL, "1")
>       field(ZRST, "MANUAL")
>       field(ONST, "AUTO")
>       field(PINI, "YES")
> }
>
> ENV_AGC_MONITOR is used with camonitor, ENV_AGC_SP is used with
> caget/caput. When I set the value with ENV_AGC_SP, ENV_AGC_MONITOR
> should refresh the value.
>
>>
>> Nevertheless - some thoughts:
>> Your NOBT is zero, but your mask is correct, probably something that
>> asyn does for you.
>> Looked at the record code: An mbbi should definitely send VAL updates
>> when it processes and the value changed.
>
> Yes it should, but this does not happen always.
>
>> Are you sure asyn gets the value you expect it to into the record?
>> (Check by enabling asyn debug.)
>
> I've checked the status of the low level stuff (kernel driver and
> hardware) and the new value is there..
>
>> Are you sure the record gets processed by asyn when you expect it to?
>> (Check by setting the trace field.)
>
> I want it to react as soon as the value in hardware changes - hence the
> I/O scanning. Hmm, maybe I could try some other Periodic scans to see if
> it helps..
>
> Will try the asyn debug and traces too..
>
>> When you set up multiple camonitor clients - do they all behave the
> same?
>
> Yes, they do.
>
> I've also tried running camonitor on several clients, even locally on
> the IOC server itself (ARM cross compiled), but without any success. It
> seems to me that EPICS TCP transport layer is not reacting to the change
> !?
>
>> When you do intermediate cagets while your camonitor is not getting
>> updates, what timestamps are these showing?
>
> They seem to be showing some old timestamp - e.g. few hours back, while
> it should show current date&time stamp.
>
>> (BTW: You only have two states with raw values 0 and 1 defined - any
>> particular reason for using mbbi as opposed to bi?)
>
> No particular reason at all.. :)
>
>>
>> Someone with more asyn background could probably say more....
>
>
> Thank you for the valuable pointers!
>
> Best regards,
> Hinko
>
>
>>
>> Good luck!
>> Ralph
>>
>>
>> On Tue 11 May 2010 10:07:12 Hinko Kocevar wrote:
>>> Hello Ralph,
>>>
>>> On 05/11/10 15:21, Ralph Lange wrote:
>>> 
>>>> Hello Hinko,
>>>>
>>>> what EPICS versions are you using for client and IOC?
>>>>    
>>>
>>> IOC and client use 3.14.10 EPICS base version.
>>>
>>> 
>>>> Can you show a dump (dbpr <name>, 4) of the record that you are
>>>> connecting to?
>>>>    
>>>
>>> Sure, here it is:
>>> epics> dbpr LIBERA01:ENV:ENV_AGC_MONITOR 4
>>> ACKS: NO_ALARM      ACKT: YES           ASG:                ASP:
>>> 0x00000000
>>> BKPT: 0x00          COSV: NO_ALARM      DESC: AGC           DISA: 0
>>>
>>> DISP: 0             DISS: NO_ALARM      DISV: 1             DPVT:
>>> 0x50c72c00
>>> DSET: 0x7c620840    DTYP: asynInt32     EIST:               EISV:
>>> NO_ALARM
>>> EIVL: 0x0           ELST:               ELSV: NO_ALARM      ELVL: 0x0
>>>
>>> EVNT: 0             FFST:               FFSV: NO_ALARM      FFVL: 0x0
>>>
>>> FLNK:CONSTANT 0     FRST:               FRSV: NO_ALARM      FRVL: 0x0
>>>
>>> FTST:               FTSV: NO_ALARM      FTVL: 0x0           FVST:
>>>
>>> FVSV: NO_ALARM      FVVL: 0x0           INP:INST_IO @asyn(ENV, 725,
> 1.0)
>>>
>>> LALM: 1             LCNT: 0             LSET: 0xe8752e00    MASK:
>>> 4294967295
>>> MLIS: 0x38d0320060d0320003000000        MLOK: 0x28700b00    MLST: 1
>>>
>>> NAME: LIBERA01:ENV:ENV_AGC_MONITOR      NIST:               NISV:
>>> NO_ALARM
>>> NIVL: 0x0           NOBT: 0             NSEV: NO_ALARM      NSTA:
>>> NO_ALARM
>>> ONST: AUTO          ONSV: NO_ALARM      ONVL: 0x1           ORAW: 1
>>>
>>> PACT: 0             PHAS: 0             PINI: YES           PPN:
>>> 0x00000000
>>> PPNR: 0x00000000    PRIO: LOW           PROC: 0             PUTF: 0
>>>
>>> RDES: 0xa8a10300    RPRO: 0             RSET: 0x34b70a40    RVAL: 1
>>>
>>> SCAN: I/O Intr      SDEF: 1             SDIS:CONSTANT       SEVR:
>>> NO_ALARM
>>> SHFT: 0             SIML:CONSTANT       SIMM: NO            SIMS:
>>> NO_ALARM
>>> SIOL:CONSTANT       SPVT: 0x388a2f00    STAT: NO_ALARM      SVAL: 0
>>>
>>> SVST:               SVSV: NO_ALARM      SVVL: 0x0           SXST:
>>>
>>> SXSV: NO_ALARM      SXVL: 0x0           TEST:               TESV:
>>> NO_ALARM
>>> TEVL: 0x0           THST:               THSV: NO_ALARM      THVL: 0x0
>>>
>>> TIME: 2010-05-11 11:18:40.100440000     TPRO: 0             TSE: 0
>>>
>>> TSEL:CONSTANT       TTST:               TTSV: NO_ALARM      TTVL: 0x0
>>>
>>> TVST:               TVSV: NO_ALARM      TVVL: 0x0           TWST:
>>>
>>> TWSV: NO_ALARM      TWVL: 0x0           UDF: 0              UNSV:
>>> NO_ALARM
>>> VAL: 1              ZRST: MANUAL        ZRSV: NO_ALARM      ZRVL: 0x0
>>>
>>>
>>> 
>>>> What field of that record are you connecting to?
>>>>    
>>>
>>> I did no specify any field - so it is VAL, I guess by default.
>>>
>>> Thank you!
>>>
>>> Regards,
>>> Hinko
>>>
>>> 
>>>> Regards,
>>>> Ralph
>>>>
>>>>
>>>> On Tue 11 May 2010 6:26:48 Hinko Kocevar wrote:
>>>>   
>>>>> Hi all,
>>>>>
>>>>> I'm experiencing some strange behavior when using camonitor against
> an
>>>>> EPICS IOC.
>>>>>
>>>>> For some reason a CA_PROTO_EVENT_ADD response from IOC to CA client
>>>>> does
>>>>> not arrive, or arrives several minutes later after the PV value has
>>>>> changed. Note that first CA_PROTO_EVENT_ADD response is seen
> always,
>>>>> only latter CA_PROTO_EVENT_ADD responses do not arrive (or arrive
>>>>> late)?!
>>>>> This behavior is not seen always - sometimes the IOC would promptly
>>>>> respond with CA_PROTO_EVENT_ADD response when needed.
>>>>>
>>>>> I can see the CA_PROTO_ECHO requests/responses fine on the same
>>>>> camonitor channel link, only CA_PROTO_EVENT_ADD response is not
> seen.
>>>>>
>>>>> Inspecting the IOC I can see that the value did change (on lower
>>>>> levels)
>>>>> and caput response is also valid - updated with new value.
>>>>>
>>>>> I've ruled out the network problems, since the issue is raised even
>>>>> when
>>>>> using cross-over cable to communicate with the IOC. Actually even I
>>>>> update the value in the IOC shell, the camonitor is not refreshed!!
>>>>>
>>>>>
>>>>>
>>>>> Has someone experienced something similar?
>>>>> Are there some conditions that need to fulfilled in order for IOC
> to
>>>>> send a subsequent CA_PROTO_EVENT_ADD response(s)?
>>>>>
>>>>>
>>>>>
>>>>> TIA & HAND,
>>>>> Hinko
>>>>>      
>>>
>>>
>>>  
>
>

--
Hinko Kocevar
Technical support software engineer
Instrumentation Technologies
Velika pot 22, SI-5250 Solkan - Slovenia
T:+386 5 3352600, F:+386 5 3352601
mailto: [email protected]

http://www.i-tech.si <http://www.i-tech.si/>  - When your users demand stability

The information transmitted is intended solely for the addressee and may
contain confidential and/or privileged information. Any review, retention,
disclosure or other use by persons other than the intended recipient is
prohibited. If you received this in error, please notify the sender and
delete all copies.

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System