Experimental Physics and
Industrial Control System

Geoff Savage <[email protected]> · Mon, 17 Apr 2006 11:27:26 -0500

Hi,
There have been a number of emails over the last few months
discussing pushing alarms from EPICS along with the DZero
implementation ("hack"?) to push alarms from EPICS IOCs to a central
server running on a Linux host computer. This has also been referred
to as opt-out alarms from EPICS. Our implementation uses a hook in
IOC core. The hook is a function written by me that gets called
every time a record is processed and starts the push process. We
added our hook to the end of recGblResetAlarms(). recGblResetAlarms
() is called from monitor(), the second to last function called in a
records' process function. The last function called, recGblFwdLink,
processes the forward scan if specified.
There was a discussion of redefining recGblSetSevr to send alarms so
that no hook was needed. Currently recGblSetSevr sets a records'
status and severity and stores the first alarm condition with the
highest severity. If an alarm was pushed out each time recGblSetSevr
is called you could end up with multiple alarms from the same
record. We did not want multiple alarms from the same record. Our
goal was to report the alarm condition for a record as determined by
EPICS.
The consequence of not having a standard hook (see initHooks as an
example) is that each version of EPICS base that we use requires me
to modify EPICS base to include the hook. We are in favor of
permanently adding a hook in base to push out alarms. If it would
help I am willing to present our implementation during the EPICS
collaboration meeting in June and I'm willing to attend any sessions
of a SIG working on this issue.
I have purposefully not commented on any other aspects of opt-out
alarm handling here. Instead I have included a brief description of
the Significant Event System which has been in use at DZero for the
last four years at the end of this email.
Your friendly neighborhood hacker, Geoff

Brief description of the Significant Event System :
EPICS provides tools for handling alarms generated from PVs,
including an operator alarm display. However, alarms from slow
controls are not the only,neither the only nor, necessarily, the most
important events in the DZero data acquisition system (DAQ). To
process events from all DAQ sources, of which slow controls is but
one source, we have developed a separate facility, the Significant
Event System (SES) [3][44], to collect and distribute all changes of
state of the detector and the data acquisition system. The SES has
a central server that collects event messages from sender clients and
filters them, via a Boolean expression, for receiving clients.
Sender clients, which include the IOCs, connect to the server via
ITC, a locally developed, inter-task communication package based upon
TCP/IP sockets. A; and all state changes on those clients, including
alarm transitions, are sent to the server. server. The client-server
model has advantages over the EPICS alarm facility, wherein which the
operator display explicitly connects to each PV. There is no need
to construct extensive configuration files for the hundreds of
thousands of PVs in the slow controls system and the savings in
connect time at startup can be significant.
The alarm class of SES messages receives special handling in the
server. The SES server maintains the current alarm state of the
entire detector so that receiving clients are able to obtain the
current state when they first connect to the server. In addition to
specialized receiving clients that may connect to the server, there
are three standard clients: the SES logger, the SES alarm display,
and the SES alarm watcher. The logger has a pass-all filter so that
it receives all SES messages sent to the server and writes the
messages received to a disk file. The current state of the detector
stored in the server is relayed to users through the alarm display.
There is a global configuration for the alarm display and the ability
to specialise the configuration for the purposes of individual sub-
detectors. For alarms that compromise data quality the alarm
watcher will automatically pause the current run.
In addition to its monitoring and logging functions, the SES system
provides the means for distributing synchronizing messages to other
components of the online software system. For example, global control
of the high-voltage system is accomplished by having the individual
detector high-voltage programs connect to the SES server for messages
that signal changes in the run state of the data acquisition system.

Software tools have been developed for mining data from the SES log
files. Hardware experts review the log files to understand which
hardware devices are unstable and . cCollaborators performing data
analysis can insure the event they are examining is real and not
caused by a fault in the detector.
The SES server and most of the receiving clients have been coded in
the Python scripting language, while many of the sending clients are
coded in C or C++. We anticipate that, for efficiency
considerations, the server may require recoding in C++ at some later
stage in the development cycle. API’s for SES clients are available
in all three languages.

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System