Experimental Physics and
Industrial Control System

"Dalesio, Leo `Bob`" <[email protected]> · Thu, 22 Jun 2006 10:58:07 -0700

This is an important point that Jeff is making and there is some policy in place - I'm sure.
How do the alarm packages work for DESY and FNAL?
For redundancy we're decided that we do not need to send every change to the backup IOC - only the most recent values of every field that has changed.
For an alarm logger, this seems less optimal. However, if the client cannot keep up, then the queue may occassionally overflow. What are the queue overflow policies for each of these alarm handlers?
Thanks,
Bob 

-----Original Message-----
From: Jeff Hill [mailto:[email protected]] 
Sent: Thursday, June 22, 2006 9:53 AM
To: Dalesio, Leo `Bob`; 'Ralph Lange'
Cc: 'Geoff Savage'; 'Matthias Clausen'; 'Andrew Johnson'; 'EPICS core-talk'; 'Fritz Bartlett'; 'Liyu, Andrei'
Subject: RE: alarm hook

In send-only-on-change subscription update systems one always needs some per subscription state that remembers that there was no queue space and that we were forced to discard a particular update. Later when queue space becomes available we will need to send an update for all subscriptions that are not current. In the CA server's event queue I handle this by replacing the last update on the queue for a particular subscription when the queue saturates.

Based on private discussions with Geoff and Andrei I fear that they are currently implementing an unreliable send-only-on-change subscription update event stream where alarm state change edges can be lost forever (if the alarm queue saturates during an alarm burst). 

The original Fermilab alarm system is, I suspect, not a send-only-on-change system, but instead a send-periodically based system and therefore might be more likely to keep clients in sync.

Anyways that is my perception surrounding recent alarm hook developments. I am fearful that the EPICS community could be straying away from reliable event transport, but feel free to realign my discernment of the situation if I am uninformed.

Jeff

> -----Original Message-----
> From: Dalesio, Leo `Bob` [mailto:[email protected]]
> Sent: Thursday, June 22, 2006 5:47 AM
> To: Ralph Lange
> Cc: Geoff Savage; Matthias Clausen; Andrew Johnson; EPICS core-talk; 
> Fritz Bartlett; Liyu, Andrei
> Subject: RE: alarm hook
> 
> I'm an optimist about the general reuse of code. It's a habit. I 
> assume that we could make a minimum set of alarm consumers that would 
> be useful for all of us.
> 
> I would also like the dbPostEvent hook for archiving. I think that a 
> plant archiver that only wants data on occasion would prefer to have 
> it pushed out - rather than connect to 60000 channels. But there needs 
> to be a discussion on that.
> 
> 
> -----Original Message-----
> From: Ralph Lange [mailto:[email protected]]
> Sent: Thursday, June 22, 2006 4:44 AM
> To: Dalesio, Leo `Bob`
> Cc: Geoff Savage; Matthias Clausen; Andrew Johnson; EPICS core-talk; 
> Fritz Bartlett; Liyu, Andrei
> Subject: Re: alarm hook
> 
> I think the answers to all your questions highly depend on what A and 
> B are, which is the part not covered by the recent discussions.
> Depending on the implementation of createMessage() A and B might be 
> Oracle servers, parts of the D0 event system, CMLOG servers .... with 
> all these different systems, I doubt there even will be a default 
> implementation for createMessage().
> The alarm viewers are clients to these A and B servers, so even the 
> specs might differ depending on the system or installation.
> 
> Ralph
> 
> 
> Dalesio, Leo `Bob` wrote:
> > Will A&B have a way to synchronize?
> > Is there thoughts of being able to serve the alarm information from
> multiple sources to many clients?
> > Any specs on the alarm viewers?
> > Thanks,
> > Bob
> >
> > -----Original Message-----
> > From: Geoff Savage [mailto:[email protected]]
> > Sent: Wednesday, June 21, 2006 10:36 PM
> > To: Matthias Clausen
> > Cc: Geoff Savage; Andrew Johnson; EPICS core-talk; Fritz Bartlett; 
> > Liyu, Andrei
> > Subject: Re: alarm hook
> >
> > Hi,
> >
> > I am willing to accept the hook as proposed by Andrew.  This is not 
> > the
> exact interface that I am currently using but it does provide the 
> information that I require.  I will modify the FNAL interface to match 
> the proposed interface once the hook is in base.
> >
> > It is not clear to me how an alarm hook function is registered in a
> startup script.  Can someone please provide an example that is 
> operating system independent?
> >
> >  From our (Matthias Clausen, Fritz Bartlett, Vladimir Sirotenko, 
> > Geoff
> Savage) discussions on Tuesday we developed a "generic" (but not 
> matching Andrei's interface) interface.  I'll address Andrei's email 
> next.  We propose to include the following "tools" for pushing alarms 
> from the server in a package separate from base.  Here is a simple 
> chain showing all the pieces.
> >
> > hook -> logAlarm -> queue -> sendToNetwork -> createMessage -> send 
> > to A -> send to B (if send to A fails)
> >
> > Here is a summary of our discussions on each piece with some of my
> experiences included (and indicated).
> >
> > 1. We start with the generic hook as proposed by Andrew.
> >
> > 2. The hook function which a user will register is logAlarm(struct
> dbCommon *prec, unsigned short sevr, unsigned short stat).  Once this 
> function has determined that an alarm needs to be sent it gathers the 
> volatile data and inserts it into the alarm data queue.  It should 
> also do something reasonable if the data can't be inserted in the 
> queue.  It might simply keep track of the number of insertion failures 
> and report this number in a message once the queue has space.
> > Geoff - It should send messages on bad and good transitions.  Care
> should be taken when a record is successfully processed for the first 
> time as it transitions from an undefined (bad) state to a good state.
> >
> > 3. Users should be able to adjust the size of this queue to 
> > accommodate
> different numbers of records in the IOCs.
> > Geoff - There are more alarms during maintenance periods than during 
> > run
> times.
> >
> > 4. The sendToNetwork function is running as a separate thread 
> > (vxworks
> task) at a priority lower than the scan tasks but higher than channel 
> access.  It waits for data to arrive in the alarm data queue.  When 
> data arrives it passes the data to the createMessage routine.
> >
> > 5. The createMessage routine constructs a string message to be sent
> across the network.  To be more generic users should be able to define 
> their own createMessage routine.  This allows users to use a different 
> message protocol within the push out framework.
> > Geoff - Using a string for messages removes byte ordering issues.
> >
> > 6. Send the message to server A.  If sending the message to server A
> fails then send the message to server B.
> >
> > All the requirements are not included here.  Hopefully DESY will 
> > provide
> more details as the project progresses.
> >
> > Some other issues to consider -
> > a. From Jeff Hill - can we detect on a record by record basis when 
> > an
> alarm is not pushed out.  I think this requires some study of the 
> common fields available to all records.  This is on my to do list.
> > b. What will be in the data inserted into the alarm data queue?  
> > This
> should be all the volatile data needed in the network message to 
> decrease the time spent collecting the data.
> > c. What are the contents of the network message?
> > d. A generic server also needs to be provided.
> >
> > Geoff
> > P.S.  I need to sleep and will reply to Andrei's email in the morning.
> >
> >
> >
> > On Jun 19, 2006, at 10:02 PM, Matthias Clausen wrote:
> >
> >
> >> Hi Andrew,
> >>
> >> I had a meeting today with Geoff, Fritz and Vladimir at Fermilab.
> >> We discussed the implementation based on the proposed function call 
> >> and agreed on an implementation which should be as generic as 
> >> possible.
> >> After Geoff's and Bernd Schoeneburg's vacation (next two weeks) we 
> >> will work on an implementation.
> >> If Geoff does not see any unforeseen problems I would like to give 
> >> you a 'go' for the change in base.
> >>
> >> Thanks for your help!
> >> And - by the way - thanks for your clarification regarding Andrei's 
> >> mail.
> >>
> >> Matthias
> >>
> >>
> >> Andrew Johnson wrote:
> >>
> >>> Hi Matthias,
> >>>
> >>> Matthias Clausen wrote:
> >>>
> >>>> in preparation for my meeting with Geoff at Fermilab I wand to 
> >>>> send you the proposed hook into base which Bernd Schoeneburg and 
> >>>> Bob already 'somehow' agreed on:
> >>>>
> >>>> Here's Berns mail to Bob:
> >>>>
> >>>>
> >>>>> recGblResetAlarms is called in monitor() which is called in the 
> >>>>> end of record processing just before recGblFwdLink and after 
> >>>>> recGblGetTimeStamp. After calling recGblResetAlarms in monitor() 
> >>>>> the value changes are checked (not interesting for us).
> >>>>> recGblResetAlarms checks for alarm changes and returns the fist 
> >>>>> approach of the monitor mask, which is later used for postEvents.
> >>>>> postEvents can be called from anywhere like device support, snl, 
> >>>>> subroutines, 'homebrew' records etc. I think recGblResetAlarms 
> >>>>> is called in the monitor function of records only. So I think it 
> >>>>> is the perfect place. Please check it and correct me if I am wrong.
> >>>>>
> >>>>> The code could look like this (the end of recGblResetAlarms):
> >>>>>
> >>>>>     if(sevr!=nsev || stat!=nsta) {
> >>>>>
> >>>>> ++:    logAlarm (pdbc, sevr, stat);
> >>>>> ++:    /* nsev and nsta are in pdbc->sevr and pdbc->stat */
> >>>>>
> >>>>>         ackt = pdbc->ackt; acks = pdbc->acks;
> >>>>>         if(!ackt || nsev>=acks){
> >>>>>             pdbc->acks=nsev;
> >>>>>             db_post_events(pdbc,&pdbc->acks,DBE_VALUE);
> >>>>>         }
> >>>>>     }
> >>>>>     return(mask);
> >>>>> }
> >>>>>
> >>>> My question:
> >>>> Do you also agree with approach?
> >>>>
> >>> I agree with the location and arguments of the call, which I 
> >>> believe are the same as Fermilab have been using.
> >>>
> >>>
> >>>> And - what would be implemented in base for logAlarm ()?
> >>>> This could be an empty function which just returns - or it could 
> >>>> be the 'real' thing where you'll have to check whether alarm 
> >>>> logging should be used at all.
> >>>> The empty function could be replaced/ overloaded by the 'real'
> >>>> function if you want to use putAlarm.
> >>>>
> >>> I think we just need a global pointer for the routine which will 
> >>> be called if it's not NULL, so your code just sets it to hook in.
> >>> Here's my proposed patch:
> >>>
> >>>
> >>> Index: recGbl.h
> >>> ==================================================================
> >>> = RCS file: 
> >>> /net/phoebus/epicsmgr/cvsroot/epics/base/src/db/recGbl.h,v
> >>> retrieving revision 1.9
> >>> diff -u -b -r1.9 recGbl.h
> >>> --- recGbl.h    12 Feb 2003 21:22:23 -0000      1.9
> >>> +++ recGbl.h    19 Jun 2006 15:25:16 -0000
> >>> @@ -30,13 +30,23 @@
> >>>       : FALSE\
> >>>  )
> >>>
> >>> +/* Structures needed for args */
> >>>
> >>> -/* Global Record Support Routines*/  struct link;  struct dbAddr; 
> >>> struct dbr_alDouble;  struct dbr_ctrlDouble;  struct dbr_grDouble;
> >>> +struct dbCommon;
> >>> +
> >>> +/* Hook Routine */
> >>> +
> >>> +typedef void (*RECGBL_ALARM_HOOK_ROUTINE)(struct dbCommon *prec,
> >>> +    unsigned short sevr, unsigned short stat); extern 
> >>> +RECGBL_ALARM_HOOK_ROUTINE recGblAlarmHook;
> >>> +
> >>> +/* Global Record Support Routines */
> >>> +
> >>>  epicsShareFunc void epicsShareAPI recGblDbaddrError(
> >>>      long status, struct dbAddr *paddr, char *pcaller_name); 
> >>> epicsShareFunc void epicsShareAPI recGblRecordError(
> >>> Index: recGbl.c
> >>> ==================================================================
> >>> = RCS file: 
> >>> /net/phoebus/epicsmgr/cvsroot/epics/base/src/db/recGbl.c,v
> >>> retrieving revision 1.60.2.2
> >>> diff -u -b -r1.60.2.2 recGbl.c
> >>> --- recGbl.c    4 Nov 2004 19:21:08 -0000       1.60.2.2
> >>> +++ recGbl.c    19 Jun 2006 15:25:16 -0000
> >>> @@ -42,6 +42,10 @@
> >>>  #include "recGbl.h"
> >>>
> >>>
> >>> +/* Hook Routines */
> >>> +
> >>> +RECGBL_ALARM_HOOK_ROUTINE recGblAlarmHook = NULL;
> >>> +
> >>>  /* local routines */
> >>>  static void getMaxRangeValues();
> >>>
> >>> @@ -239,6 +243,7 @@
> >>>      if(stat_mask)
> >>>          db_post_events(pdbc,&pdbc->stat,stat_mask);
> >>>      if(sevr!=nsev || stat!=nsta) {
> >>> +       if (recGblAlarmHook) (*recGblAlarmHook)(pdbc, sevr, stat);
> >>>         ackt = pdbc->ackt; acks = pdbc->acks;
> >>>         if(!ackt || nsev>=acks){
> >>>             pdbc->acks=nsev;
> >>>
> >>>
> >>> If there is general agreement between DESY and FNAL about this, 
> >>> I'll commit the change which will then appear in R3-14-9.
> >>>
> >>> - Andrew
> >>>
> >> --
> >> -------------------------------------------------------------------
> >> --
> >> -
> >> --
> >> Matthias Clausen                         Cryogenic Controls Group
> >> (MKS-2)
> >> phone:  +49-40-8998-3256                Deutsches Elektronen
> >> Synchrotron
> >> fax:    +49-40-8994-3256
> >> Notkestr. 85
> >> e-mail: [email protected]                           22607
> >> Hamburg
> >> WWW-MKS2.desy.de
> >> Germany
> >> -------------------------------------------------------------------
> >> --
> >> -
> >> --
> >>
> >>
> >
> >

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System