Experimental Physics and
Industrial Control System

"Jeff Hill" <[email protected]> · Fri, 4 Mar 2005 16:26:07 -0700

Ken,

This sounds pretty serious. Its hopefully a rare situation where
the wrong value actually gets written to hardware by EPICS. An
entry in Mantis is probably appropriate. 

Since they are reproducing this with both EDM and DM, then I
doubt that this is an OPI problem. They may have discovered this
because they "lost" a device. If that's the case then we may be
looking for a bug where the gateway is forwarding one spurious
corrupted value which is rapidly replaced by a correct value, and
that situation might not be routinely detected unless it caused
something to fail. 

For debugging the issue, perhaps it would be best to concentrate
on setting up a small system that will reproduce this (perhaps
with only a soft IOC, a gateway, and MEDM). To reliably detect
the problem we may need to add some database logic that latches
an out of range value condition so that we can detect transient
anomalies. If that can be done then Ken, or I, should be able to
quickly isolate the cause in the debugger. It may help to be
running with a debugger that can break on a particular value
ending up in a variable (as is the case with many Intel based
debuggers).

Rolf,

o I assume that the gist of reproducing this will be to attach an
OPI on the outside of the gateway, rapidly yank the slider back
and forth, and see if a bad value ends up in the record or
device? Is this something that can be caused, on demand, by an
operator's choice of wrist action, or is the problem happening
once in a blue moon but suspected to be related to slider use. Do
you know for certain that the bad value does not occur until the
slider is released. Or does it occur also when the gateway has
backlog and then catches back up even if the slider was once very
busy, is now inactive, but is not yet released. 

o Your seeing this only when going through the gateway probably
implicates the gateway. Nevertheless, there is also some chance
that this is a device driver issue and that the gateway times
things differently enough to uncover a device driver related race
condition. Has this been seen with more than one type of device
(or record)? Is the device involved using the asynchronous IO
features of the device support interface? Are multiple clients
writing to the same device at the same time? Does the device
driver have mutex locking in it to protect against writes from
multiple threads?

O If you would like to accelerate the debugging process then you
might try to reproduce this on a small system there (perhaps with
only a soft IOC, a gateway, and DM/EDM/MEDM) and send us explicit
instructions on what you did.

Jeff

> -----Original Message-----
> From: Rolf Keitel [mailto:[email protected]] 
> Sent: Friday, March 04, 2005 12:15 PM
> To: tech_talk
> Subject: dm and edm problems with CA gateway
> 
> 
> **** Has anybody else experienced these problems? ****
> 
> When we are operating sliders to write process variables 
> through the CA 
> gateway, every now and then on releasing the sliders the 
> gateway writes 
> huge (and I mean   r e a l l y   huge positive or negative) 
> numbers to 
> the IOC, which set the PV to the upper or lower end of the 
> allowed range.
> This never happens if you don't use the gateway.
> 
> We observed this behaviour both with dm and with edm (running 
> on Solaris 
> or Linux Redhat 9).
> The gateway is running on a Linux (Redhat 9) box. Several
gateway 
> releases up to Beta18 (built against EPICS R3.14.6) exhibit 
> this behaviour.
> 
> We lost a couple of devices, where a user hadn't specified a 
> proper DRVH 
> for the setpoint.
> We have taken the gateway out of operation until this is
resolved.
> 
> - rolf -
> 
> -- 
> +-------------------------------------------------+
> | Rolf Keitel, TRIUMF         Tel: (604) 222-7453 |
> | Group Leader ISAC Controls  Fax: (604) 222-7307 |
> | and Electronics Development                     |
> | Vancouver, B.C., Canada                         |
> +-------------------------------------------------+
> 

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System