CAT Chat Minutes

July 21, 2003

 

Information and Follow-up

Randy Alkire stated that the intercom messages in the sector 19 wet lab were inaudible.

EFOG (Ray) found that the volume control for the lab intercom was set low. There is a volume control for each LOM lab located within the lab. EFOG (Patti) has placed labels on all the controls "Intercom Volume Control".

User asked about the wireless network connection to visitors network (Violet) that has been made available in APS. Are there plans to bring this capability to the ring?

Equipment has been ordered for hi-bay installation. The schedule for installation depends on arrival date of hardware. This sytem utilizes the 80211-B server which will be upgradeable to the higher speed 80211-G server when it becomes available at the end of this year.

 

General Information

Operation Stats R. Klaffky:

Delivered beam:  1025 hours
Percentage of schedule time: 94.22%
Downtime during period: 62.89 hours
Mean time between faults (MTBF):  31.07 hours

The fault information follows:

# 55 07/16 08:00 To 07/16 11:46 3.78

13 BM PSS trip [ES]

0.51 Refill

# 56 07/16 12:17 To 07/17 15:49 27.54

S23AV3/RTFB problem [Diagnostics/ES]

3.82 Lost again due to power bump[PFS],recover and refill

# 58 07/17 19:39 To 07/19 02:36 30.95

MPS card failure [CTL]

2.88 Diagnose, Refill

# 59 07/19 05:28 To 07/19 12:24 6.93

S22:23 Raw Supply [ES]

0.54 Reset, Refill

# 60 07/19 12:57 To 07/19 21:23 8.43

Network hub card failure and back-up failure [ISS]

1.00 Diagnose, Refill

# 61 07/19 22:23 To 07/20 20:56 22.56

S17FBIOC reboot [OPS]

0.30 Refill

# 62 07/20 21:14 To 07/21 02:45 5.50

Lightning strike [OTHER]

0.50 Refill

# 63 07/21 03:15 To 07/21 10:27 7.20

MPEOB intlk problem [ES]

2.14 Repair, magnet conditioning, refill

Beamline Downtime R. Klaffky:  Roger will present downtime figures at the next CAT Chat.

User asked what is a network storm. Ken Sidorowicz explained that Epics, the accelerator control software, uses broadcast packets to locate PV's on the ioc's. The uplinks for hubs r12b failed which isolated bpm, pss, and id ioc's in sectors 11 through 19 from the rest of the accelerator network. Because these ioc's were unavailable the workstations in the MCR were sending out broadcast packets in an attempt to find the missing PV's. A "network storm" is when the rate of broadcast packets increases to a level that network performance is degraded. This occurred on Saturday but was not the reason for the downtime on the accelerator. The fact that some ioc's in a quarter of the ring were inaccessible is the actual cause.

Roger commented that there would be a meeting to discuss lessons learned, what went wrong and how we can prevent future problems.