E000P-921170-00
Date: 01/10/00
E000P-921170-00
01/10/00
SERIOUS FAULT RESETTING PROCEDURES
for the
ADVANCED PHOTON SOURCE
PERSONNEL SAFETY SYSTEM
of the
ARGONNE NATIONAL LABORATORY
APPROVED BY:
Jon Hawkins, Date
ISI Group Leader, AOD
PREPARED BY:
John Stoffel, Date
ISIG Hardware Section Leader, AOD
REVIEWED BY:
__________________
Nick Friedman, Date
ISI Interlock Systems Section Leader, AOD
__________________
Roy Emerson, Date
ISIG Software Section Leader, AOD
__________________
Bob Ferry, Date
Experimental Floor Operations, AOD
TABLE OF CONTENTS
1 INTRODUCTION.................................................................................................................................................................. 5
1.1 Purpose and Objectives........................................................................................................................................ 5
1.2 Scope................................................................................................................................................................................ 5
1.3 Definitions................................................................................................................................................................... 5
1.4 Applicability.............................................................................................................................................................. 5
2 RELATED DOCUMENTS.................................................................................................................................................... 5
2.1 Non-Government Documents........................................................................................................................... 5
2.2 Precedence................................................................................................................................................................... 5
3 PROCEDURE......................................................................................................................................................................... 5
3.1 PSS Minor Fault Reset Procedure for a validated beamline PSS............................................. 5
3.2 PSS Serious Fault Reset Procedure for a validated beamline PSS.......................................... 6
3.3 PSS Major Fault Reset Procedure for a validated beamline PSS............................................. 9
Table of Figures
Serious Shutter Fault/Minor DIW Fault Reset Flow Diagram
REVISION RECORD
|
Revision |
Dated |
Comments |
|
0 |
02/21/95 |
Initial relapse |
|
1 |
01/10/00 |
Change Section 3 to include more complete statement on Serious PSS Fault Reset Procedure |
|
|
|
|
|
|
|
|
|
|
|
|
1 INTRODUCTION
1.1 Purpose and Objectives
The purpose of this document is to assure that only authorized personnel perform the fault reset activity, safety concerns are identified, and that this trouble is properly documented.
1.2 Scope
This document includes all fault reset operations to be performed on a validated Personnel Safety System (PSS). It is recommended that this document be reviewed every 1 - 2 years.
1.3 Definitions
The definition of “fault reset operations” on a validated PSS is any time a Minor, Serious or Major Fault is reset on a validated and operational PSS.
1.4 Applicability
1.4.1 PSS Authorized Personnel
Only ISIG personnel or Floor Coordinators that are trained in the operating of the PSS shall perform fault reset operations on a validated PSS.
2 RELATED DOCUMENTS
The following documents form a part of this specification to the extent specified herein.
2.1 Non-Government Documents
2.1.1 ANL/APS/AOD Specification and Interface Control Documents
Work Request Procedures and Forms E000P-921140
2.2 Precedence
In the event of conflict between the provisions of this specification and other documents, the following precedence shall apply; the latest specification applies.
3 PROCEDURE
3.1 PSS Minor Fault Reset Procedure for a validated beamline PSS
3.1.1 Record, via AOD Trouble Report, non-redundant PSS minor fault number, time/date stamp, conditions when fault occurred. This information is usually obtained from the EPICS PSS screens and USER information.
3.1.2 Perform PSS minor fault reset by either floor Coordinator (FC) or ISIG member.
3.1.3 If the minor fault resets then continue with normal operation.
3.1.4 If minor fault does not reset then contact the On-Call ISIG staff person.
3.1.5 The ISIG On-Call staff person will manage the PSS minor fault problem.
3.1.6 If work on the PSS is needed then follow the process described in the "PSS Work Order Procedures" E000P-921140
3.2 PSS Serious Fault Reset Procedure for a validated beamline PSS
3.2.1 Record, via AOD Trouble Report, the PSS Serious Fault number, time/date stamp, and conditions when fault occurred. This information is usually obtained from the EPICS Log and/or APS Users
3.2.2 The following A - G procedure is to determine if the Serious faults are caused by either shutter blow-by or DIW flow/pressure (see Appendix A for flow chart of this Serious fault procedure):
A) At the present time MAJOR faults are not candidates for RESET. The SERIOUS Faults that can be RESET by the Floor Coordinators are Shutter related (NO SWITCH and MIXUP).
B) The first step for the Shutter faults is to determine that the Front-End and Beamline shutters closed properly by accessing the EPICS Beamline PSS Inputs Page or noting the Front End Shutter indicator showing RED (closed position) on the USER Panel.
To access the PSS Inputs page, start at XFD Page (XFD-Display.adl) and select EPS/PSS then select PSS Menu. This is PSS_Overview.adl where we select the Beamline (e.g. 14-ID) and then select Inputs
This page is PSS/new/Inputs_14ID.adl (where 14ID is the Beamline identifier). Check that both of the Shutter’s Chain A and Chain B closed limit switches are closed except as noted.
Front End Shutter Limit Switches are IPS2LS, ISS1LS, & ISS2LS.
Beamline Integral Shutter Limit Switches are IP5MS1 & IP5MS2 (Also P8 and P9).
Beamline Mode Shutter Limit Switches are IP4PS, IP4SS, IP4MS1, & IP4MS2 (Also P6). It should be noted that on a Mode Shutter in White/Pink mode, MS1 and MS2 are locked in the OPEN position with PS and SS energized in the OPEN position.
C) If any of the Front End Shutters (PS2, SS1, & SS2) don’t show all of their Closed Limit Switches closed, we don’t want to open a Station door or go Global Off-Line since this will cause a Major fault and a Main Ring trip.
If any of the Beamline Shutters (except the noted Mode Shutter) don’t show all of their Closed Limit Switches closed and the Front End Shutters are properly closed, we can only have Serious Faults which won’t trip the Main Ring hence we can go Off-Line and work on the shutter.
In either case with improper Limit Switch indications, call the PSS Staff On-Call person.
D) If the Limit Switches are indicating properly, there are two situations to consider:
1) NO SWITCH and MIXUP faults on Shutters that occur when they are being opened. This is usually caused by the blow-by problem.
If the above closed status switches indicate properly, reset the Serious fault and try to open the shutter(s).
If the attempt was successful, check the inputs EPICS page for proper open indications for both A and B Chains.
If the attempt was unsuccessful, note the time for the fault to indicate (should be 10 seconds) and the open status switches on the EPICS PSS inputs page (did either the A or B switches show open?).
If the shutter fails for three consecutive attempts to open or there is any doubt about the situation, call the BLOPS staff ON-CALL person to consult for further action. A BLOPS tech may be called in to work on the shutter.
E) NO SWITCH and MIXUP faults on Shutter(s) that are already open could be due to loss of a Chain B DIW cooling flow permit.
These permits can be seen on the Inputs EPICS page. These DIW inputs (With A and B Chain inputs for each) are INONMONO AFLOW, INONMONO BFLOW, INONMONO CFLOW, and INONMOMO DFLOW. Where A, B, C, and D refer to the Stations being protected.
Loss of the Chain A input will cause a PSS DIW Minor fault while loss of the Chain B DIW input will close a Shutter(s) due to loss of the Chain B Permit usually resulting in a Serious Shutter(s) fault but no PSS DIW fault.
These permits come from the Love Controllers in the DIW chassis (which is detailed on accompanying information sheets) in the PSS rack on the mezzanine. If there is no Chain B permit (due to water flow problems) for a shutter, the corresponding Love Controller is latched and should be in the alarm state. The upper (red) display shows the Flow or Differential Pressure being monitored. The lower (green) display shows the alarm point except in the case of 1-ID which reads 0.00 but has labels with the alarm points.
F) For the DIW situation with the Shutter(s) Closed Limit Switch information and the DIW numbers, call the BLOPS tech ON-CALL person. Usually there are three possibilities.
If the Alarm point on the Controller needs to be adjusted, open the PSS rack and access the secondary menu of the Love Controller by simultaneously pressing the Up Arrow and ENTER keys. Use the INDEX key to access ALLo (Low Alarm) and ALHi (High Alarm) then press the UP or DOWN ARROW to change the Alarm settings. After reaching the new Alarm Point, press the ENTER key to store it and make it active.
These alarm points are listed on a info sheet on the PSS rack door and on a spreadsheet titled PSS WATER (DIW) COOLED DEVICES.
After this work is complete and the DIW flows look normal (no readings close to an alarm point), open the PSS rack and reset the Controller by simultaneously pressing the INDEX and DOWN ARROW keys for 5 seconds.
Then check the system by closing the main flow valve to the station and observing that the Love Controllers trip at the new flow and pressure levels. Reset again and log this information.
G) With the Shutters or DIW working properly, reset the Serious Fault(s) and reopen the shutters.
3.2.3 If the cause of the Serious Fault does not match the above description of shutter Blow By or DIW faults then call the PSS Staff ON-CALL person.
3.2.4 After understanding the causes of the Serious fault the ISIG on-call staff person will approve resetting the PSS Serious Fault. This may be done by either a Floor Coordinator or and ISIG Member.
3.2.5 If the PSS Serious fault does not reset and if work on the PSS is needed then a ISIG On-Call Staff person will follow the process described in the "PSS Work Order Procedures" E000P-921140.
3.2.6 Submit the initial portion of the TR. The URL address is http://www.aps.anl.gov/xfd/operations/bin/trouble_report.cgi
3.3 PSS Major Fault Reset Procedure for a validated beamline PSS
3.3.1 Record, via AOD Trouble Report, PSS major fault number, time/date stamp, conditions when fault occurred. This information is usually obtained from the EPICS screens and/or APS Users
3.3.2 During SR running periods, take the affected PSS Global Off-Line ASAP so normal SR operations may be restored then contact the ISIG On-Call staff person.
3.3.3 Currently all PSS major faults must be reset by ISIG members.
3.3.4 If the major fault does not reset then determine if immediate work is needed/possible on the affected PSS. Before any work is done on any validated PSS the PSS Work Procedure E000P-921140 must be followed.
3.3.5 If the major fault resets then fully record the fault event and consider resuming normal operation after understanding the causes of the major fault with approval of the PSS system manager or designee.
3.3.6 Submit the initial portion of the TR. The URL address is http://www.aps.anl.gov/xfd/operations/bin/trouble_report.cgi
Figure 1