EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

2002  2003  2004  2005  2006  <20072008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 2002  2003  2004  2005  2006  <20072008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: diagnostics related to discarding an intermediate CA subscription update
From: Claude Saunders <[email protected]>
To: Jeff Hill <[email protected]>
Cc: "'Ernest L. Williams Jr.'" <[email protected]>, "'EPICS core-talk'" <[email protected]>, EPICS-tech-talk <[email protected]>, "'Kay-Uwe Kasemir'" <[email protected]>
Date: Fri, 26 Jan 2007 14:13:48 -0600
Regarding the issue of "flow control" diagnostics, would it be reasonable to approach this like telecom and server manafacturers? I am suggesting using SNMP to offer these diagnostics much like how we monitor dropped packets on commercial routers. It is UDP-based, so would probably have little impact on IOC performance, especially if only polled from NMS clients slowly and at low priority. Like Jeff pointed out, I wouldn't be inclined to send flow controls warnings up the same pipe that is already backed up. I consider "flow control" diagnostics something that is better suited to out-of-band communication: out-of-band in the sense of either a separate protocol, or even a separate NIC.

We could register an EPICS IOC MIB and all that...

 - Claude

Jeff Hill wrote:
Ernest,

Jeff, is there a way to address the request for diagnostic info about CA
flow control.  Kay has asked about this one previously?  Would this be a
minor or major change?  We want to be able to see when flow control
"kicks in"

First there are two situations that can cause an intermediate subscription
update to be discarded.
1) Event queue entry quota saturation
2) Client library initiated flow control

Currently only the server, and not the client library, knows about the
discard of an intermediate updates due to event queue entry quota
saturation. Furthermore, event queue entry quota saturation is a
per-channel, and not a per-circuit phenomenon. In contrast, flow control is
a per-circuit phenomenon which is initiated by the client library when it
knows that it, or the application, isn't keeping up. However similarly, both
mechanisms are characterized by the number of updates that are discarded in
the server. For example, the client can be in flow control mode, but if
there isn't more than one update while in this mode we will not discard an
update.
It would of course be unwise to print a mandatory message on the server's or
client's console when discarding an intermediate subscription update because
printf is much higher overhead call compared to the per subscription update
message cost in CA so this could be another "adding more fuel to the fire
once it starts" situation that we would need to avoid.

On the server side, we already have the dbel (<record name>, <interest
level>) diagnostic which at sufficiently high interest levels reports the
number of discards that have occurred for each subscription attached to a
particular record.

For the client to know about the magnitude of the problem we would need for
the serve to send a message to the client. It makes no sense to inject a
message into the protocol stream when there is a discard as this could be
another "adding more fuel to the fire once it starts" situation that we
would need to avoid. Perhaps the sensible thing to do would be to include a
number of discards since the last update count in the update message. This
would require a free slot in the message header (we unfortunately don't have
one) or a change in the format of the message payload. Changing the format
of the payload is possible, but we would need to send the modified payload
only to clients of a particular protocol version range. This, and any
changes required to convey the information to the client's callback, might
be considered to be a large change for a point release such as R3.14.9. We
would also slightly increase the per-update message size, and therefore
potently reduce maximum throughput, but that might tend to be a negligible
impact.

So I am not predisposed to provide the type of functionality described in
the proceeding paragraph in R3.14.9, but if there appears to be a general
level of interest I am very willing to create a Mantis entry against R3.15
summarizing a feature request for that release which could be brought up for
consideration at a future EPICS developers meeting.

Jeff

-----Original Message-----
From: Ernest L. Williams Jr. [mailto:[email protected]]
Sent: Thursday, January 25, 2007 2:15 PM
To: Jeff Hill
Cc: [email protected]; 'EPICS core-talk'
Subject: Re: FW: R3.14.9: Any changes still pending?

On Thu, 2007-01-25 at 14:01 -0700, Jeff Hill wrote:
Jeff - are all your fixes committed now?  I don't want to go ahead
with
the -RC1 tag until I get a positive response from you.
I have no patches pending at this time.
Jeff, is there a way to address the request for diagnostic info about CA
flow control.  Kay has asked about this one previously?  Would this be a
minor or major change?  We want to be able to see when flow control
"kicks in"


Thanks,
Ernest



Jeff

-----Original Message-----
From: Andrew Johnson [mailto:[email protected]]
Sent: Thursday, January 25, 2007 11:18 AM
To: EPICS core-talk
Subject: R3.14.9: Any changes still pending?

Does anyone have any changes for R3.14.9 that they haven't committed
yet?  I would like to tag the -RC1 release tomorrow, but my rules
imply
that there's no point doing that if someone has any patches to commit.

I have a minor IOC shutdown issue on Linux that I should have pushed
harder with earlier, but the solution would affect many source files
and
we don't have time to discuss the fix now so I'll document it on the
Known Problems page and leave the resolution until later.

I am also still in the process of running the mrkSoftTest suite.

Jeff - are all your fixes committed now?  I don't want to go ahead
with
the -RC1 tag until I get a positive response from you.

- Andrew
--
The right to be heard does not automatically include
the right to be taken seriously. -- Hubert H. Humphrey



References:
FW: R3.14.9: Any changes still pending? Jeff Hill
Re: FW: R3.14.9: Any changes still pending? Ernest L. Williams Jr.
diagnostics related to discarding an intermediate CA subscription update Jeff Hill

Navigate by Date:
Prev: Re: FW: R3.14.9: Any changes still pending? Ernest L. Williams Jr.
Next: Proposal for an addition to configure/tools/convertRelease.pl Benjamin Franksen
Index: 2002  2003  2004  2005  2006  <20072008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: diagnostics related to discarding an intermediate CA subscription update Jeff Hill
Next: Proposal for an addition to configure/tools/convertRelease.pl Benjamin Franksen
Index: 2002  2003  2004  2005  2006  <20072008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 02 Feb 2012 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·