Experimental Physics and
Industrial Control System

"Lawrence T. Hoff" <[email protected]> · Mon, 06 Feb 2006 13:33:53 -0500


	I can recent share BNL experience, but it does *not* include
EPICS per se. Maybe it will be helpful anyway...
	The BNL IT division ran network security scans last month (I
*think* NESSUS is the tool of choice). These tests are run from
ITD-managed platforms which are *not* on the "accelerator network".
ITD prefers to use their own platforms for their convenience and
control. This means that we had to agree to support such scans by
"opening up" the "accelerator network" firewall. We would never agree
to such scans during operational periods.
	Most of our systems experienced what amounts to a DOS
attack during the scans, but returned to pre-scan behavior once
the scans were over. For instances, while probing TELNET on our
VxWorks (5.5.1) systems, ITD was able to cause TELNET server
tasks to get suspended. However, VxWorks eventually cleanup
those tasks up. However, two classes of systems had lingering
problems:
	1) Many of our terminal servers needed to be rebooted to
"free up" TCP/IP ports. I *guess* the TCP/IP connections were formed
using a "SYN attack", or some other pathological protocol. The result
was the terminal servers thought the ports were in use, and would
not allow other users. Even days later the ports were still unavailable.
	2) Some of our MV162 VxWorks systems lost network operation,
and needed to be rebooted to restore network operation. This is a bug
which we have seen under "heavy" network load in the past, and which
has been discussed on tech-talk (IIRC). We have learned a distinctive
signature for this situation... It is impossible to "ping" such a
system from elsewhere, and it it impossible to "ping" elsewhere
from an affected system. Using a terminal server (assuming the TS is
working!), systems which are "broken" have a characteristic response to
the "sys596Port" command:
-> sys596Port
sys596Port
interrupt: ei driver: command field frozen
0xfcd894 (tExcTask): 12 messages from interrupt level lost.
value = 0 = 0x0

	My personal conclusion is that *if* a determined hacker was
able to penetrate our "accelerator network" firewall, *and* launch a
DOS attack against all network-connected devices, such an attack
would have a crippling effect on accelerator operations. In general,
systems are tuned and configured to assume a certain level of network
availability and functionality, and when that level is compromised,
system performance is compromised. Our approach if a inside-the-firewall
DOS attack is launched is to isolate and eliminate the attack (vs.
trying to design the control system to continue to operate well
during such an attack).
	As I said before, ITD network scans require our complicity (to
reconfigure the F/W). Our working assumption is that during such scans,
system performance is not guaranteed. After such scans, we expect to
reboot all terminal servers and all VxWorks systems, just to be safe.
We do not expect to need to reboot any desktop systems.
	Finally, we perform periodic "network dependency tests", where
we unplug the "accelerator network" from the rest of the world (by
actually disconnecting a cable from the firewall), to make sure that
the control system continues to operate while completely isolated.
This helps to ensure that the control system is not affected by a DOS a
attack against an "outside" resource that the control system may depend
on.
HTH -- Larry

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System