Experimental Physics and
Industrial Control System

"Jeff Hill" <[email protected]> · Wed, 10 Nov 2010 13:47:19 -0700

Hi Tim,

Some general comments on this issue.

1) Fundamental queuing theory says that if the producer, the database, is producing events at an average rate even slightly faster than the consumption rate of the consumer, the ca client, then an unbounded amount of buffer space will be consumed if we do not discard intermediate events. Unbounded is a bad design of course in an IOC context.

2) If we allow discarding of intermediate events on a subscription then we can't always guarantee ordered delivery when comparing one subscription update to another.

3) As Ralph has mentioned the legacy rsrv server has a somewhat quirkier behavior when it starts to discard intermediate events in terms of which subscription gets discarded and what impact that has on other subscription sharing the same event block - an internal data structure of the legacy event queue. The rsrv event queue is quite efficient having been designed to run on 20MHz processors with 1MB of RAM. 

4) The portable (PCAS) server library and the new server library both have completely redesigned event queues that run slightly less efficiently, but also do not suffer from the quirkiness in (3). Nevertheless you can't in any implementation get away from fundamentals; see comments (1) and (2) above.

5) The design in (4) is certainly better, but this doesnât make that much difference for client design; clients still need to be robustly designed to handle situations where intermediate events might be discarded.

Jeff
______________________________________________________
Jeffrey O. Hill           Email        [email protected]
LANL MS H820              Voice        505 665 1831
Los Alamos NM 87545 USA   FAX          505 665 5107

Message content: TSPA

With sufficient thrust, pigs fly just fine. However, this is
not necessarily a good idea. It is hard to be sure where they
are going to land, and it could be dangerous sitting under them
as they fly overhead. -- RFC 1925

> -----Original Message-----
> From: [email protected] [mailto:tech-talk-
> [email protected]] On Behalf Of Tim Mooney
> Sent: Tuesday, November 09, 2010 1:26 PM
> To: [email protected]
> Subject: Re: monitors received out of order
> 
> I have some new data on this problem:
> 
> My client has 16 queue blocks.  The PVs whose monitors are getting time
> reversed are assigned to blocks 4 and 7, with the PV showing up early
> assigned to block 4.  This fits the scenario Ralph outlined below.  (By
> the way, I've verified that flowControlMode was never on during event
> posting or event reading.)
> 
> For concreteness, here's the apparent time sequence that's causing
> trouble:
>      record A posts DATA=0 to queue block 7
>      record A causes record B to process
>      record B posts DATA=0 to queue block 4
>      event_task gets the CPU and calls the monitor routine with record
> B.DATA,
>          then it calls the routine with record A.DATA.  My client
> thinks A.DATA
>          was posted after B.DATA.
> 
> Under vxWorks, this doesn't happen, I suppose because the event task
> always gets the CPU in the time between the two postings.  ("Always" is
> pretty strong, but the consequence of time-reversed monitors in this
> case
> is obvious enough and serious enough that I would have been notified,
> and the software has been running for many years, on many IOCs.)
> 
> I have a work-around, but it's not very robust.  If I call ca_add_event
> (ca_create_subscription) for the time-sensitive PVs *last*, most of
> them
> get assigned to the same event-queue block, and the problem goes away.
> This is partly luck, of course, because it doesn't guarantee that all
> the
> PVs will be in the same block; it just makes it more likely.
> 
> I could ensure that the PVs go in the same block by using inside
> information
> from dbEvent.c, but that doesn't seem like a very good idea.  I could
> also ensure
> it by using a separate CA context for the time-sensitive PV's, but I
> would lose the
> time ordering between these PVs and the others.  I'm pretty sure that
> would cause
> other problems.  I could implement a handshake between the monitor
> routine
> and the record, so that record A waited for the monitor to arrive
> before causing
> record B to process, but that would be a little messy, and also add
> overhead that
> a good same-block solution would not add.  I'm not sure what's the best
> way to go.
> 
> Tim
> 
> On 11/1/2010 4:27 PM, Ralph Lange wrote:
> >  On 01.11.2010 17:09, Tim Mooney wrote:
> >> Dear folks,
> >>
> >> I have two records, and a separate task monitoring a field that both
> records
> >> post, all in the same IOC.  Most of the time, my task receives
> monitors in the
> >> order in which they were posted.  But sometimes, the task receives a
> monitor
> >> from record B before it receives a previously posted monitor from
> record A.
> >> (I know for sure which record is posting first, because record A
> posts before
> >> causing record B to process.  Also, I've modified the record to set
> its time
> >> stamp immediately before posting, and I get another time stamp on
> entry to
> >> the monitor routine.  The time ordering of those stamps does not
> agree.)
> >> I've seen this on solaris and Linux, but not on vxWorks.
> >>
> >> I have code that misbehaves when this happens, so I started digging
> around
> >> and have convinced myself that I should not be relying on the time
> ordering of
> >> monitors received from different records (even when those records
> are running
> >> in the same task).  I now think I can rely on posts from a single
> record
> >> arriving in
> >> time order, but not posts from different records.  I think this
> because
> >> events from
> >> different record go into different queues, and there doesn't seem to
> be any code
> >> in the vicinity that seems worried about time ordering across event
> queues.
> >>
> >> Am I right about this?
> >
> > Short answer: yes.
> >
> > The event queue is allocated in chunks. Each chunk has room for a
> number of
> > subscriptions, and holds the values that are on the queue for these
> > subscriptions.
> > The event task (per client) simply loops over the event queue chunks
> for this
> > client (lines ~940 in dbEvent.c), and calls event_read() for each
> chunk, which
> > loops over the queue entries in a chunk (line ~815) and pushes
> everything it
> > finds there to the CA server (via user_sub, line ~865).
> > Depending on which subscriptions from which records are held in which
> event
> > queue chunk, the order on the network may vary.
> >
> > You are almost correct: The only order that you can rely on is the
> order of
> > updates for a single subscription. (Two subscriptions for different
> fields of
> > the same record may end up in different event queue chunks and get
> sent
> > out-of-order.)
> >
> > Ralph
> >
> 
> --
> Tim Mooney ([email protected]) (630)252-5417
> Software Services Group, Advanced Photon Source, Argonne National Lab.

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System