CAT Chat Minutes

October 22, 2007

Operations Update, Mark Beno

There was one beam interruption this week, a water pump failed because of a breaker problem. This resulted in 1.13 hours of down time.

There were no drops in the top up current.

Beamline Problems:

  • 22 BM and ID network problems
  • 23-ID Vacuum leak in shutter
  • 10-ID ID control problem
  • 23-ID BLEPS problem

Stats:

  • 99.25% - Beamtime
  • 85.75 hrs. Mean time between faults

Safety: Tom Barkolow

TMS and Training Anyone having a registered domain as USR-APS must have a sector person assigned as the supervisor. Please get this information to Rachel Reed ( rreed@aps.anl.gov ).

Chemical Management Assessment: Paul Rossi

The chemical management assessment will take place in Bldg. 433 and Bldg. 436 on October 23 rd at 1:00 p.m.

Temporary Electrical Installations: Jim Lang

An ANL-hired contractor has been assigned to conduct an OSHA type inspection. Some items to be inspected include the use of power cords for permanent use and temporary wiring. This inspection is being done to determine long/short term plans as well as eliminate problems in the future.

ESAF, Hazard Classes for Nanomaterials: Bruce Glagola

Additions to the ESAF will include the following hazard classes for Nano:

  • Solid Matrix (Low Risk)
  • Liquid Suspension (Medium Risk)
  • Unbound (High Risk)

A new column will be added to the ESAF materials table. When implemented, nanomaterials will have:

  • Hazard class, HC14
  • Others will be HC15

If working with Nanomaterials:

  • Review ESH manual and the hazard class
  • Add explosives column to the materials table

Training stats are under 8%. Great job!

Firewall Problems: Ken Sidorowicz

The details pertaining to the firewall problems during the week of Oct 15 th are as follows:

  • Started on Oct 15 th at 11:32 am. The CPU load on both nodes of the firewall started to spike very quickly.
  • The firewall vendor Secure Computing was contacted at 11:42 am
  • Iota and Zeta firewalls crashed.
  • Both firewalls up and running properly at 12:47pm.  Worked with Secure Computing on issue and call was escalated to engineering to review core dump and audit logs.
  • 4:45pm - Both firewalls started to CPU spike again.
  • Oct 16 th CPU load spiking continues. Dave Leibfritz added script to monitor load every 5 minutes for potential overload.
  • Oct 17 th Occasional CPU load spiking continues. Iota rebooted at 1:30pm and 4:30pm. We were placed on Secure Computing hot list. CEO was notified.
  • Oct 18 th Iota rebooted at 5:30am. Both firewalls have been stable since this time.
  • The call is at the highest priority at Secure Computing.
  • The APS Firewall cluster was installed in September 2005 for the CATs. Oct 15 th , 2007 is the first time both firewalls have been down since installation.

Comments:

The APS annual evacuation drill will take place tomorrow, October 23 rd . The minimum distance away from the building must be 1500 ft. Please follow the instructions of the Area Emergency Supervisors.

Next Meeting:

The next CAT Chat meeting will be held on October 2o, 2007.