EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  <20102011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  <20102011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: monitors received out of order
From: Tim Mooney <[email protected]>
To: [email protected]
Date: Tue, 09 Nov 2010 14:26:07 -0600
I have some new data on this problem:

My client has 16 queue blocks.  The PVs whose monitors are getting time
reversed are assigned to blocks 4 and 7, with the PV showing up early
assigned to block 4.  This fits the scenario Ralph outlined below.  (By
the way, I've verified that flowControlMode was never on during event
posting or event reading.)

For concreteness, here's the apparent time sequence that's causing trouble:
    record A posts DATA=0 to queue block 7
    record A causes record B to process
    record B posts DATA=0 to queue block 4
    event_task gets the CPU and calls the monitor routine with record B.DATA,
        then it calls the routine with record A.DATA.  My client thinks A.DATA
        was posted after B.DATA.

Under vxWorks, this doesn't happen, I suppose because the event task
always gets the CPU in the time between the two postings.  ("Always" is
pretty strong, but the consequence of time-reversed monitors in this case
is obvious enough and serious enough that I would have been notified,
and the software has been running for many years, on many IOCs.)

I have a work-around, but it's not very robust.  If I call ca_add_event
(ca_create_subscription) for the time-sensitive PVs *last*, most of them
get assigned to the same event-queue block, and the problem goes away.
This is partly luck, of course, because it doesn't guarantee that all the
PVs will be in the same block; it just makes it more likely.

I could ensure that the PVs go in the same block by using inside information
from dbEvent.c, but that doesn't seem like a very good idea.  I could also ensure
it by using a separate CA context for the time-sensitive PV's, but I would lose the
time ordering between these PVs and the others.  I'm pretty sure that would cause
other problems.  I could implement a handshake between the monitor routine
and the record, so that record A waited for the monitor to arrive before causing
record B to process, but that would be a little messy, and also add overhead that
a good same-block solution would not add.  I'm not sure what's the best way to go.

Tim

On 11/1/2010 4:27 PM, Ralph Lange wrote:
On 01.11.2010 17:09, Tim Mooney wrote:
Dear folks,

I have two records, and a separate task monitoring a field that both records
post, all in the same IOC.  Most of the time, my task receives monitors in the
order in which they were posted.  But sometimes, the task receives a monitor
from record B before it receives a previously posted monitor from record A.
(I know for sure which record is posting first, because record A posts before
causing record B to process.  Also, I've modified the record to set its time
stamp immediately before posting, and I get another time stamp on entry to
the monitor routine.  The time ordering of those stamps does not agree.)
I've seen this on solaris and Linux, but not on vxWorks.

I have code that misbehaves when this happens, so I started digging around
and have convinced myself that I should not be relying on the time ordering of
monitors received from different records (even when those records are running
in the same task). I now think I can rely on posts from a single record arriving in
time order, but not posts from different records. I think this because events from
different record go into different queues, and there doesn't seem to be any code
in the vicinity that seems worried about time ordering across event queues.


Am I right about this?

Short answer: yes.


The event queue is allocated in chunks. Each chunk has room for a number of subscriptions, and holds the values that are on the queue for these subscriptions.
The event task (per client) simply loops over the event queue chunks for this client (lines ~940 in dbEvent.c), and calls event_read() for each chunk, which loops over the queue entries in a chunk (line ~815) and pushes everything it finds there to the CA server (via user_sub, line ~865).
Depending on which subscriptions from which records are held in which event queue chunk, the order on the network may vary.


You are almost correct: The only order that you can rely on is the order of updates for a single subscription. (Two subscriptions for different fields of the same record may end up in different event queue chunks and get sent out-of-order.)

Ralph


-- Tim Mooney ([email protected]) (630)252-5417 Software Services Group, Advanced Photon Source, Argonne National Lab.


Replies:
RE: monitors received out of order Jeff Hill
References:
monitors received out of order Tim Mooney
Re: monitors received out of order Ralph Lange

Navigate by Date:
Prev: Re: EPICS Suggestions and Priorities Andrew Johnson
Next: Re: EPICS Suggestions and Priorities emmanuel_mayssat
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  <20102011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: monitors received out of order Ralph Lange
Next: RE: monitors received out of order Jeff Hill
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  <20102011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 10 Nov 2010 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·