EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  <20062007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  <20062007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Motion control failure at APS
From: Ron Sluiter <[email protected]>
To: EPICS <[email protected]>
Date: Mon, 13 Nov 2006 12:12:07 -0600

On Tuesday, July 25, a serious motion control failure occurred in the 34ID-A hutch here at the APS.

All eight stepper motors controlled by a single Oregon
Micro Systems (OMS) VME58 board ran past their limit
switches and into their mechanical stops.

The eight stepper motors involved in the accident are
used to position two white beam slits and a mirror.

Bellows up stream from the 1st white beam slit were
punctured by the x ray beam.  Vacuum was lost.
Additional damage included two broken limit switches
and an ion pump.

Three days of beamtime were lost.

Like all APS beamlines using the OMS motor controllers,
this IOC relies on the OMS controller to stop motion
when a motor is moving in the direction of an activated
limit switch.  In this case, not only did the OMS
controller fail to stop motion when the limit switches
were activated, the controller entered a strange failure
mode that drove the motors into their mechanical stops.

The VME crate in question is located inside the 34ID-A
white-beam hutch. It is possible that radiation damage
contributed to the failure of the OMS VME58 controller.

There are hundreds of OMS boards at APS that have been in
operation for many years, and no report of this type of
failure has ever been made.  OMS has also never had
a report of this type of failure.

Between Kurt Goetze, OMS and myself; we have determined
that the failure resulted from two characteristics of
the OMS VME58 board, that, together conspired to cause
this failure.  First, we have known for some time that
the OMS VME58 boards output a step pulse on every axis
after a VMEbus reset.  Second, (unknown to us) the
on-board VME58 watchdog can cause the board to go into
a continuous reset cycle if there is a board failure.

These two characteristics, together, caused the failed
VME58 board at 34ID-A to step all eight of its' motors at
approximately 4 steps per second until power was shut off.

Since this accident OMS has resolved the "step pulse on
reset" problem.  By fixing the "step pulse on reset" problem,
OMS prevents the VME58 from accidentally driving its' motors
if a board fails and exhibits the "continuous reset" behavior.

The "fix" requires the replacement of nine socketed
components on the VME58 controller board.

OMS is providing two ways to retrofit VME58 boards with
this fix.

Option #1 - Send your VME58 boards back to OMS.  They will
replace the nine components and retest the board.  To do
this, first contact OMS for a Return Materials Authorization
(RMA) number and refer to Engineering Waver #112 (EW112)
as the reason for the RMA. Cost is $357.00 per board.

Option #2 - OMS is providing a retrofit kit that includes
the nine parts (5 of which are pre-programmed), labeling,
instructions and shipment for $107 per kit.

I don't have the specifics on how to order the kit.  I will
post that info. in a follow-up message.

If you are ordering new OMS VME58 boards, be aware that OMS
will not be incorporating the upgrade chip set unless you
specify Engineering Waiver no. 112 (EW112) in your order.
In addition, we recommend the MAXv rather than the VME58
for new orders.

For those who choose to upgrade existing VME58 boards,
I strongly advise option #1.  Option #2 opens you to the
risk of taking a perfectly functioning VME58 board and
making it less reliable; e.g., bending a pin while inserting
a part in a socket or damage from static discharge. Option
#1 provides full testing and quality assurance from OMS.

Closing thoughts;

- Thanks to OMS for addressing and fixing this problem.
The VME58 product family is approaching it's end of life.
The only incentive for OMS to devote resources to resolving
this problem is their commitment to quality.

- The "step on reset" problem does not occur in OMS's
latest controller; the MAXv and MAXp controllers.

- OMS VME58 user's should review what the consequences of
having this failure occur at their facilities would be.
The decision to have your boards retrofitted should strike
a balance between the probability of board failure (higher
if exposed to radiation) and the consequences of that
failure.

Sorry for the length; hope I didn't bore you,
Ron


Replies:
RE: Motion control failure at APS - Not a Problem for Hytec 8601 / 8909 Darrell Nineham

Navigate by Date:
Prev: devSNMP: Thread error? Bertrand H.J. Biritz
Next: RE: Motion control failure at APS Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  <20062007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: devSNMP: Thread error? Bertrand H.J. Biritz
Next: RE: Motion control failure at APS - Not a Problem for Hytec 8601 / 8909 Darrell Nineham
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  <20062007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 02 Sep 2010 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·