EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: FW: Fatal seq_monitor: pvVarMonitorOff
From: Benjamin Franksen <[email protected]>
To: Janez Golob <[email protected]>, <[email protected]>
Date: Thu, 10 Oct 2013 15:06:33 +0200
Hi Janez

On Thursday, October 10, 2013 08:19:11 you wrote:
> Please find attached log file. I've updated seqencer module to use
> errlogPrintf instead of printf functions. This allows me to use ioc
> logger. If the output goes to the console the problem doesn't appear
> since my board uses quite a low baud rate. Of course most of the
> messages are discarded but you will see two of them at line 207 and
> 212.

Thanks, this confirms my suspicion that the failure is due to a race
condition in the sequencer.

I will provide a fix as soon as possible.

For the meantime, here are some alternatives to avoid the problem. You
need to change your program anyway, see point (3) below.

(1) It is generally not necessary to call pvMonitor after pvAssign. It
is simpler (and incidentally avoids the bug in the sequencer) to call
pvMonitor only once *before* the pvAssign. Or, even simpler, you could
use a top-level monitor clause for the whole array:

	monitor setpoint_reached;

The run time system remembers the monitor on/off state of each variable
and when re-connecting automatically sets up monitor subscriptions.

(2) You should be aware that if pvAssign is called for a variable that
has an active connection to a PV, the existing connection will be closed
and a new one opened. This is not very efficient. The log file you sent
me shows that this is what happens (a lot), there are lots of lines like

	10.5.2.234:53072 Thu Oct 10 08:02:41 2013 0xde21750:
pvVariable::~pvVariable()

You could avoid that by checking the connection status with pvConnected
before using pvAssign. (But see (3) for a better solution.)

(3) The way you are handling re-connects is logically wrong, and this is
completely independent of the sequencer bug.

Let me repeat what you sent in your original message, formatted in a
more compact and readable way:

	state initialize {
		when (pvConnectCount() == pvAssignCount()) {
			pvGet(mag_count, SYNC);
			for (i = 0; i < mag_count; i++) {
				epicsSnprintf(recordName, sizeof(recordName),
					"MAGDIPL%d:IS_SETPOINT_REACHED", i);
			}
			pvAssign(setpoint_reached[i], recordName);
			pvMonitor(setpoint_reached[i]);
			/*epicsThreadSleep(3.0);*/
		} state wait_start
		when (delay(10)) {
		} state initialize
	}
	state wait_start {
		when (pvConnectCount() < pvAssignCount()) {
		} state initialize
		when (efTestAndClear(pws_start_flag)) {
		} state magnets_to_zero_amps
	}

Since pvAssign does not wait for the connection callback, when state
wait_start is entered, the condition pvConnectCount() < pvAssignCount()
will almost never be TRUE (except when un-commenting the call to
epicsThreadSleep), which means you immediately go back to state
initialize. *This* is where the program actually waits for the
connections to be established. Then, as soon as this is the case, you
again call pvAssign, causing channels to be closed and re-opened, and
everything starts again.

I hope it is clear now that this will be an "endless" loop. It might
accidentally happen that the loop gets broken when --due to artefacts of
scheduling and/or system load-- the task gets held up long enough so
that the connections really are established when state wait_start is
entered.

If I have guessed correctly what your program is *supposed* to do, a
better solution would be:

monitor setpoint_reached; /* flag all elements as "to-be-monitored" */
...
ss ... {
	state initialize {
		when (pvConnected(mag_count)) {
			/* the only connection we need at this point */
			pvGet(mag_count, SYNC);
			for (i = 0; i < mag_count; i++) {
				/* this assumes setpoint_reached[i] is assigned to "" */
				epicsSnprintf(recordName, sizeof(recordName),
					"MAGDIPL%d:IS_SETPOINT_REACHED", i);
			}
			pvAssign(setpoint_reached[i], recordName);
		} state wait_connected
	}
	state wait_connected {
		when (pvConnectCount() == pvAssignCount()) {
		} state wait_start
	}
	state wait_start {
		when (pvConnectCount() < pvAssignCount()) {
			/* note: channels will automatically re-connect */
		} state wait_connected
		when (efTestAndClear(pws_start_flag)) {
		} state magnets_to_zero_amps
	}

If you run this with "debug=1" you will see that you get a *lot* less
messages and especially all the destructor calls disappear.

(Disclaimer: code not tested, there may be typos and/or stupid errors).

Cheers
Ben

> -----Izvirno sporočilo-----
> Od: Benjamin Franksen [mailto:[email protected]]
> Poslano: 9. oktober 2013 15:54
> Za: Janez Golob
> Zadeva: Re: FW: Fatal seq_monitor: pvVarMonitorOff
>
> On Wednesday, October 09, 2013 14:45:32 Janez Golob wrote:
> > When running a sequncer program on the vxWorks IOC I've noticed the
> > following errors being reported:
> > .
> >
> > sevr�seq_monitor: pvVarMonitorOff(var setpoint_reached[5], pv
> > MAGDIPL5: IS_SETPOINT_REACHED) failure:
> >
> > sevr�seq_monitor: pvVarMonitorOff(var setpoint_reached[6], pv
>
> > MAGDIPL6: IS_SETPOINT_REACHED) failure:
> Could you please run your program (the failing version) with "debug=1"
> as (extra) run-time parameter and send me the complete output (as an
> attachment) up to teh point where the error message appears?
>
> I have an idea what causes the problem but I want to understand it
> better before I fix it and/or recommend a work-around.
--
"Make it so they have to reboot after every typo." ― Scott Adams

Attachment: signature.asc
Description: This is a digitally signed message part.


Replies:
Re: FW: Fatal seq_monitor: pvVarMonitorOff Benjamin Franksen
References:
FW: Fatal seq_monitor: pvVarMonitorOff Janez Golob

Navigate by Date:
Prev: RE: PTP/MTP/libgphoto2 camera triggering + EPICS? Mark Rivers
Next: Re: FW: Fatal seq_monitor: pvVarMonitorOff Benjamin Franksen
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: FW: Fatal seq_monitor: pvVarMonitorOff Janez Golob
Next: Re: FW: Fatal seq_monitor: pvVarMonitorOff Benjamin Franksen
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 20 Apr 2015 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·