Please see below.
Ralph Lange wrote:
Surprising, isn't it?
As Jeff pointed out:
On the IOC (with the rsrv server), the effects that Jeff and I
described can both independently cause updates to a single client have
out of order timestamps. Here are the scenarios:
QueueChunks:
Client has many subscriptions, so there are two event queue chunks, A
and B. Many subscriptions are handled by A, some by B. Records are
happily pushing updates for their subscriptions on the event queue
(into both chunks), the last update happens to be for a subscription
that is handled in chunk A. Now the event task comes along and empties
the queues from both chunks, always starting with A.
The last update (with the newest time stamp) from chunk A is sent to
the client before the updates from chunk B (having older time stamps).
Caching mode:
Client has connections that are all in one event queue chunk. One
record is processing very fast, filling the event queue faster than
the low priority event task can take things off the queue and onto the
network. In that situation the event queue chunk switches to cache
mode, where updates are always overwriting the last queued update of
their subscription in the event queue chunk. The event queue is
implemented as a circular buffer, the data is overwritten in place.
(Pulling one element out and putting it on the end is a very expensive
operation on a ring buffer.) The last update overwrites a queue
element somewhere in the middle of the ring buffer. Now the event task
comes along and empties the queue, starting at the head of the queue.
The last update (with the newest time stamp) is sent to the client
before the following updates in the ring buffer (having older time
stamps).
So - if you think about it, under the following conditions the updates
will be in order:
1. The client subscribes to the "triggering" channel last, after it
subscribed to all other channels in that record set. (So that, even if
subscriptions are handled in different event queue chunks, the
"triggering" channel subscription will be in the same or a later chunk
than the other subscriptions.)
2. Record processing is slow enough and the network and IOC loads are
spacious enough that the event queue never switches to caching mode.
Chances are that these conditions were always true in your systems, so
your clients always saw things in the right order.
There would be random cases when data becomes out of order
because the timing passed beyond the capability of
a non real-time operating system. Those cases have nothing
to do with EPICS, but the O.S. The "out of order" did happen
to one of my projects, which is implemented under Linux.
The application does not require EPICS for the client/server.
Since EPICS is not needed, it is more efficient to wrap around
the code to implement the order. The software was coded around
some proprietary Fiber Channel protocol, which I had to sign an
NDA with a company for not releasing it.
I just did a quick check: The code in dbEvent.c has been working that
way since 1991.
So probably your trainer back then was simplifying thins a bit ... at
least that's what I do in trainings.
dbEvent.c is well written. It is written for client/server
applications. For a client/server application, the above
mentioned "out of order" would be more efficient, and more deterministic
to be solved on a real-time operating system to ensure the order of data.
Cheers,
Kate
Cheers,
Ralph
On 02.11.2010 05:26, [email protected] wrote:
Hi All,
I read Tim's initial post with interest because I am sure that I was
told at an early EPICS training course (probably in 1992 or 93) that
monitors posted to a given client within a given lockset were
guaranteed to be received in order (and, conversely, ca_puts from a
client were delivered in order). I have since used this for a lot of
systems that need to collect a coherent block of information. I have
not known it to fail, but I haven't looked for the 1 in a million
problem, just the 1 in 100 or so. Now this old wives tale is clearly
being challenged.
I find Jeff's response ambiguous, and maybe that's the problem. Jeff
says there is a per client event queue. Tim seems to imply it is
per-subscription. Ralf seems to imply the truth is somewhere in
between, and since it is more specific it has the ring of truth about
it.
Was my memory ever true? I have a suspicion that by the tone of his
message Tim may have been labouring under the same misapprehension.
It may have changed, for example, when we lifted the fixed (read 16k)
CA limit. I would agree with Tim in that in my experience it is true
most of the time - especially on vxWorks, and since I have mostly
used it to get synchronised hard real-time data from a server to a
client, vxWorks is where I have used it.
I am not sure what to do about it now, but I definitely have a
frequent requirement to get a coherent set of data from a server. Up
till now, I have monitored everything and then taken a snapshot
whenever I got a monitor from the last item in a processing chain
(which I have ensured always updates whenever the chain processes).
This has seemed to work with VxWorks servers. I would like to still
be able to achieve the same effect. How else can I do this?
Cheers,
Nick Rees
Principal Software Engineer Phone: +44 (0)1235-778430
Diamond Light Source Fax: +44 (0)1235-446713
-----Original Message-----
From: [email protected]
[mailto:[email protected]] On Behalf Of Jeff Hill
Sent: 01 November 2010 21:42
To: 'Tim Mooney'; 'EPICS tech-talk'
Subject: RE: monitors received out of order
Hi Tim,
The Channel Access publish and subscribe system is a classic
consumer/producer situation. The producer is of course record
processing and the consumer is the Channel Access client side
application. In basic queuing theory, it is known that no
matter how big the finite buffer capacity between the producer
and the consumer, all of it will be consumed if the production
rate is even a marginal amount faster than the consumption rate.
Furthermore, to prevent one client from disrupting subscription
updates sent to another client, each client attaching to an IOC
is serviced out of a per-client dedicated event queue connecting
the higher priority database processing threads with the lower
priority per-client server threads. So to avoid consuming all
available memory this event queue is finite length, and if full
capacity is reached the last event on the queue for a particular
subscription is replaced; this has the net impact of causing an
intermediate subscription update to be discarded.
So one can easily see, given the above design with intermediate
events being discarded when the producer out-paces the consumer,
that events will sometimes be delivered out of order when comparing
different channel subscriptions, but that events will never be
delivered out of order within a single channel subscription.
HTH
Jeff
______________________________________________________
Jeffrey O. Hill Email [email protected]
LANL MS H820 Voice 505 665 1831
Los Alamos NM 87545 USA FAX 505 665 5107
Message content: TSPA
With sufficient thrust, pigs fly just fine. However, this is
not necessarily a good idea. It is hard to be sure where they
are going to land, and it could be dangerous sitting under them
as they fly overhead. -- RFC 1925
-----Original Message-----
From: [email protected]
[mailto:[email protected]]
On Behalf Of Tim Mooney
Sent: Monday, November 01, 2010 3:10 PM
To: EPICS tech-talk
Subject: monitors received out of order
Dear folks,
I have two records, and a separate task monitoring a field that both
records
post, all in the same IOC. Most of the time, my task receives
monitors in
the
order in which they were posted. But sometimes, the task receives a
monitor
from record B before it receives a previously posted monitor from
record
A.
(I know for sure which record is posting first, because record A posts
before
causing record B to process. Also, I've modified the record to set its
time
stamp immediately before posting, and I get another time stamp on
entry to
the monitor routine. The time ordering of those stamps does not
agree.)
I've seen this on solaris and Linux, but not on vxWorks.
I have code that misbehaves when this happens, so I started digging
around
and have convinced myself that I should not be relying on the time
ordering of
monitors received from different records (even when those records are
running
in the same task). I now think I can rely on posts from a single
record
arriving in
time order, but not posts from different records. I think this because
events from
different record go into different queues, and there doesn't seem to be
any code
in the vicinity that seems worried about time ordering across event
queues.
Am I right about this?
--
Tim Mooney ([email protected]) (630)252-5417
Software Services Group, Advanced Photon Source, Argonne National Lab.
- References:
- monitors received out of order Tim Mooney
- RE: monitors received out of order Jeff Hill
- RE: monitors received out of order nick.rees
- Re: monitors received out of order Ralph Lange
- Navigate by Date:
- Prev:
Re: Options for reviving old Allen Bradley PLC implementations Maren Purves
- Next:
build streamDevice 2-4 error Ding Jianguo
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
<2010>
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
- Navigate by Thread:
- Prev:
Re: monitors received out of order Ralph Lange
- Next:
Re: monitors received out of order Ben Franksen
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
<2010>
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
|