Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  <20062007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  <20062007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017 
<== Date ==> <== Thread ==>

Subject: pushing alarms from epics at dzero
From: Geoff Savage <savage@fnal.gov>
To: tech-talk@aps.anl.gov
Cc: Geoff Savage <savage@fnal.gov>
Date: Mon, 17 Apr 2006 11:27:26 -0500
Hi,

There have been a number of emails over the last few months discussing pushing alarms from EPICS along with the DZero implementation ("hack"?) to push alarms from EPICS IOCs to a central server running on a Linux host computer. This has also been referred to as opt-out alarms from EPICS. Our implementation uses a hook in IOC core. The hook is a function written by me that gets called every time a record is processed and starts the push process. We added our hook to the end of recGblResetAlarms(). recGblResetAlarms () is called from monitor(), the second to last function called in a records' process function. The last function called, recGblFwdLink, processes the forward scan if specified.

There was a discussion of redefining recGblSetSevr to send alarms so that no hook was needed. Currently recGblSetSevr sets a records' status and severity and stores the first alarm condition with the highest severity. If an alarm was pushed out each time recGblSetSevr is called you could end up with multiple alarms from the same record. We did not want multiple alarms from the same record. Our goal was to report the alarm condition for a record as determined by EPICS.

The consequence of not having a standard hook (see initHooks as an example) is that each version of EPICS base that we use requires me to modify EPICS base to include the hook. We are in favor of permanently adding a hook in base to push out alarms. If it would help I am willing to present our implementation during the EPICS collaboration meeting in June and I'm willing to attend any sessions of a SIG working on this issue.

I have purposefully not commented on any other aspects of opt-out alarm handling here. Instead I have included a brief description of the Significant Event System which has been in use at DZero for the last four years at the end of this email.

Your friendly neighborhood hacker, Geoff


Brief description of the Significant Event System :


EPICS provides tools for handling alarms generated from PVs, including an operator alarm display. However, alarms from slow controls are not the only,neither the only nor, necessarily, the most important events in the DZero data acquisition system (DAQ). To process events from all DAQ sources, of which slow controls is but one source, we have developed a separate facility, the Significant Event System (SES) [3][44], to collect and distribute all changes of state of the detector and the data acquisition system. The SES has a central server that collects event messages from sender clients and filters them, via a Boolean expression, for receiving clients. Sender clients, which include the IOCs, connect to the server via ITC, a locally developed, inter-task communication package based upon TCP/IP sockets. A; and all state changes on those clients, including alarm transitions, are sent to the server. server. The client-server model has advantages over the EPICS alarm facility, wherein which the operator display explicitly connects to each PV. There is no need to construct extensive configuration files for the hundreds of thousands of PVs in the slow controls system and the savings in connect time at startup can be significant.

The alarm class of SES messages receives special handling in the server. The SES server maintains the current alarm state of the entire detector so that receiving clients are able to obtain the current state when they first connect to the server. In addition to specialized receiving clients that may connect to the server, there are three standard clients: the SES logger, the SES alarm display, and the SES alarm watcher. The logger has a pass-all filter so that it receives all SES messages sent to the server and writes the messages received to a disk file. The current state of the detector stored in the server is relayed to users through the alarm display. There is a global configuration for the alarm display and the ability to specialise the configuration for the purposes of individual sub- detectors. For alarms that compromise data quality the alarm watcher will automatically pause the current run.

In addition to its monitoring and logging functions, the SES system provides the means for distributing synchronizing messages to other components of the online software system. For example, global control of the high-voltage system is accomplished by having the individual detector high-voltage programs connect to the SES server for messages that signal changes in the run state of the data acquisition system.
Software tools have been developed for mining data from the SES log files. Hardware experts review the log files to understand which hardware devices are unstable and . cCollaborators performing data analysis can insure the event they are examining is real and not caused by a fault in the detector.


The SES server and most of the receiving clients have been coded in the Python scripting language, while many of the sending clients are coded in C or C++. We anticipate that, for efficiency considerations, the server may require recoding in C++ at some later stage in the development cycle. API’s for SES clients are available in all three languages.



Navigate by Date:
Prev: CAJ 1.0.4 issues Chu, Paul C.
Next: ASYN R4-5 available Eric Norum
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  <20062007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017 
Navigate by Thread:
Prev: CAJ 1.0.4 issues Chu, Paul C.
Next: ASYN R4-5 available Eric Norum
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  <20062007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017 
ANJ, 02 Sep 2010 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·