Experimental Physics and
Industrial Control System

"Liyu, Andrei" <[email protected]> · Mon, 22 Aug 2005 15:59:42 -0400

Jeff,

	Does record has one queue? Or PV channel has queue? Do scalar
and array be similar queue(=1)?
	When Monitor() function is called data will be copied to network
stack? Or Monitor() sets a "semaphore" to notify server layer about new
data?

	I can add one more behavior feature. Let say IOC had 20
detectors. All time BLM thread sends new data in the same order to Epics
(this is in the same IOC). Let's say PV1, PV2, ..., PV20. Clearly, PC
doesn't get data and all time it will be the tail of order, for example,
from PV15 to PV20. I see 6 lines (one per PV) that data was lost.
	Finally, all time I see one additional message that PV15 (first
lost PV) came second time (=twice with the same serial number). I
couldn't understand why I see this just one time? It should be 6 times.
Or CA has buffer for a semaphore and the semaphore is once per all
arrays?
	Or is there another source of the problem?

Thanks, Andrei.

-----Original Message-----
From: Jeff Hill [mailto:[email protected]] 
Sent: Monday, August 22, 2005 11:50 AM
To: Liyu, Andrei; [email protected]
Subject: RE: CA loss package on 1 Hz rate

Andrei,

> PC checks serial number and prints on terminal port if data was 
> lost or come twice. Time to time (very seldom) I read both problems.

Within an EPICS IOC the scan tasks run at relatively high priority and
the
CA server runs at relatively low priority. This design ensures proper
independence of scan tasks from introduced client load.

When a record is processed, and it needs to provide a subscription
update to
the CA server, it places the update in a queue. When the CA server gets
an
opportunity to run it reads any events that might be waiting on the
queue
and sends them over the wire to the CA clients that are subscribed.

To protect the IOC, there is a quota limiting the maximum length of the
CA
subscription update event queue. When this quota is exceeded the last
event
on the queue for a particular subscription is replaced. This means that
if
there is a (possibly brief) period where the scan tasks produce
subscription
updates at a rate that exceeds the rate at which the server can send
subscription updates to the client or the rate at which the client
processes
subscription updates then the buffering capacity of the event queue can
be
exceeded. The bursty nature of CPU and network loads can make this
situation
more likely to occur. CPU and network load reporting diagnostics tend to
report aggregate loads averaged over time, and do not tend to report
load
burst statistics. That would require a strip chart or a histogram.

Furthermore, the memory management for the CA event queue is currently
optimized for use with scalar value / time stamp / alarm status tuple
updates. This means that multiple value updates might be queued together
at
once allowing intermediate values to not be discarded during brief
periods
of client or network load saturation. In contrast, array value / time
stamp
/ alarm status tuples are not queued by the system. The quota for these
updates is exactly one update, and therefore it is very likely that
intermediate values can be discarded during brief intervals where the
client
CPU load or network load saturate. This behavior is by design. We were
very
careful to not call malloc/free for queuing of random sized arrays
because
of concerns about excessive CPU loading and fragmentation of pool. Worse
yet, since array element values are not placed on the queue then, with
arrays, the event is only a signal that an update is required. The
actual
value that is sent is whatever that might be in the database at the time
that the server reads the place holder array update event from the
queue.
This means that the same update value might be sent twice (as you have
observed).

The above is a well known limitation of the system at this time. With
EPICS
V4 we intend to eliminate this problem by transferring event queue
memory
management responsibility to the event producer. 

Jeff

> -----Original Message-----
> From: Liyu, Andrei [mailto:[email protected]]
> Sent: Monday, August 22, 2005 8:43 AM
> To: [email protected]
> Subject: CA loss package on 1 Hz rate
> 
> Hi,
> 
> 	SNS BLM system has 2 levels. First level is VME based/vxWorks
> (IOC). Second level is PC/Windows (PC) - similar CAGateway. IOC sends
> data one time per second to PC. Data is arrays (one array (720 floats)
> per detector; up to 32 detectors per IOC). Array has information about
> serial number of current package. PC checks serial number and prints
on
> terminal port if data was lost or come twice. Time to time (very
seldom)
> I read both problems.
> 	SNS works on 1 Hz now. IOC CPU loading ~ 3-5% and PC CPU loading
> 5%. On 60 Hz IOC will have 30% on 60 Hz. PC doesn't depend of
> accelerator rate but depends from clients quantity.
> 	I hoped to use CAArchive to find source of problem. But I met
> another problem. Archive lost connection to IOC when IOC/PC connection
> was Ok.
> 
> 	Does anyone see this problem? Any other suggestions to find the
> source (IOC, network, PC)?
> 
> Thanks, Andrei.

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System