Abstract
This paper presents the current status of the Advanced Photon Source (APS), its control system and the Experimental Physics and Industrial Control System (EPICS) tools being used to implement this control system. The status of the physical plant and each of the accelerators as well as detailed descriptions of the software tools used to build the accelerator control system are presented. The control system uses high-performance graphic workstations and the X-windows graphical user interface (GUI) at the operator interface level. It connects to VME/VXI-based microprocessors at the field level using TCP/IP protocols over high-performance networks. This strategy assures the flexibility and expansibility of the control system. A defined interface between the system components will allow the system to evolve with the direct addition of future, improved equipment and new capabilities.
The Advanced Photon Source will be a state-of-the-art facility dedicated to the production of x-ray beams. Creation of these beams begins at an electron gun. The energy of the electrons is then raised to 200 MeV by the electron linac, a high-current machine whose 30-ns beam of 1.7 Amperes impinges on a converter target, creating electron-positron pairs. The positrons are then focused through a pulsed solenoidal magnetic field into a 450-MeV positron linac. These positrons are then delivered to the positron accumulator ring (PAR). Each linac pulse generates approximately 1.5 x 109 positrons to be delivered to the PAR. Since the desired number of positrons in a storage ring bunch is much more than this, up to twenty-four linac pulses are generated in rapid succession (at a 60-Hz rate) and accumulated in the PAR
The PAR is a DC storage ring that receives the bunches of 450-MeV positrons (1.5 x 109/pulse) at a 60-Hz rate from the positron linac. The positrons are accumulated at a first harmonic frequency of 9.8 MHz during the first 0.4 second of the booster synchrotron cycle. During the last 0.1 second of the cycle, the 12th harmonic cavity is switched on (117 MHz) for final compression of the accumulated bunch and subsequent extraction at a 2-Hz rate to the booster/injector synchrotron.
The synchrorton is a 368 m ring where the energy of the positrons is raised to 7 GeV. The positrons are then injected into the storage ring located inside the experiment hall.
The storage ring is 1104 m in circumfunance an is divided into 40 sectors, 34 of which are available for experiments. Each experiment sector consists of two photon beamlines that extend tangentially from the ring inside the experiment hall. One of the two beamlines in each sector will originate at a bending magnet, which produces radiation that exhibits broadly distributed energy. The other of the beamline pairs will each begin at an insertion device (ID). These IDs can be either wigglers or undulators. Wigglers produce very intense, energetic radiation over a wide range of energies, while undulators yield radiation of selected energy at high brilliance. These tuned x-ray beams are further processed by optical instrumentation before they illuminate the sample being studied. When the APS begins operations in 1995, 16 sectors will be available to users.
II. Civil Construction Status
The physical construction of the APS began on June 4, 1990. The predominant structure is the experiment hall, which encloses the storage ring; the other buildings are either located within the infield of the experiment hall or appended to the outer perimeter of the experiment hall. The exception to this close attachment of buildings is the utility building which is located some distance from the experiment hall in order to reduce the ground motion-induced vibrations from the many pumps, chillers, compressors, and motors required to supply utilities for the facility. The machine support structures which house the house the concrete-shielded beam enclosures and technical facilities for acceleration and containment of the beam are located in the infield of the storage ring. They consist of the linac building, the synchrotron injection building, the synchrotron enclosure, the rf/extraction building, and the storage ring enclosure (located within the experiment hall). The laboratory-office modules (LOMs), are comprised of eight similar single-story buildings each located around the outer perimeter of the experiment hall. A LOM consists of laboratories, offices, conference rooms, and support spaces and provides direct access to the experiment hall.
It is expected that by December 31, 1993 all the technical buildings will be completed. The laboratory-office modules are to be completed by December 31, 1994 and the central laboratory-office building by September 1, 1995. The first users are expected in June of 1995.
The APS accelerator control system is a distributed system consisting of operator interfaces, a network, and interfaces to hardware. This implementation of a control system has come to be called the "Standard Model."
The operator interface is a UNIX-based workstation with an X-windows graphical user interface. The workstation may be located at any point on the facility network and maintain full functionality.
The function of the network is to provide a generalized communication path between the host computers, operator workstations, input/output crates, and other hardware that comprise the control system.
The crate or input/output controller (IOC) provides direct control and input/output interfaces for each accelerator subsystem.
Design challenges for the group implementing the APS controls included protecting investment from rapid obsolescence and designing into the system the ability to quickly incorporate improved equipment and software. The APS approach to solving these problems is to use a distributed control system and standards, to provide "tools" to the greatest extent possible instead of custom programming, and to collaborate with other laboratories and groups whenever possible.
The electron gun and the first two sectors of the linac have been successfully tested. The control system installation in the linac is complete and has been in use from the start of commissioning. Approximately 120 operator screens have been developed to provide the linac operations crew with information on all of the installed equipment. Present plans call for completing linac commissioning by December 31, 1993. The linac control system contains eight input/output controllers (IOCs), five of these are housed in VXI crates and the remaining three use VME crates. Allen-Bradley (AB) I/O modules are used extensively in the linac controls system. The AB I/O modules are supported by EPICS through AB scanner modules mounted in both the VXI and VME crates.
All subnets have been installed and tested. Connections to the IOCs is fiber-based Ethernet using redundant hub and transceiver technology. The linac timing system installation and testing is complete. Local consoles, implemented with X-terminals, are now operational in the klystron gallery. File servers and OPIs have been installed in the injector control room to support linac commissioning. An FDDI link between the Building 362 development area and the injector control room has been installed and is operational. This allows continued controls development and supports network growth into the final APS site.
The initial commissioning of the positron accumulator ring (PAR) will begin in January of 1994. Construction is rapidly proceeding after some initial delays in the procurement of the dipoles.
Installation of control system components for the PAR has begun and will be complete by the beginning of December. The PAR control system will use seven VME crates and four VXI crates. The VXI crates are interfaced using MXI bus technology to the IOCs located in VME crates. Allen-Bradley interfaces are used extensively to obtain data from the rf systems and to monitor the power supply and magnet cooling water. Bitbus is used to control the magnet power supplies which are based on G-64 bus technology.
The synchrotron installation is approximately 50% complete. It is expected that commissioning of the synchrotron will begin in June 1994. The synchrotron control system will consist of 11 VME IOCs, 10 VXI IOCs, and 5 Allen-Bradley I/O chassis. The IOCs will be housed in two equipment rooms located on opposite sides of the synchrotron.
The storage ring system installations will start in January 1994. Two complete sectors of the storage ring are scheduled to be operating by January 1995 to test the injection process. There are approximately 30 crates used in the storage ring control system with plans for about 50 additional crates toi control insertion devices.
The APS control system software has been constructed using the Experimental Physics and Industrial Control System (EPICS) toolkit.
EPICS is a set of software tools and applications jointly developed by Argonne National Laboratory and Los Alamos National Laboratory for the purpose of controlling particle accelerators and large experiments. Present and future development is being done cooperatively by Argonne (ANL), Los Alamos National Laboratory (LANL), Lawrence Berkeley Laboratory (LBL), the Superconducting Super Collider Laboratory (SSCL) and the Continuous Electron Beam Accelerator Facility (CEBAF). The details of the EPICS collaboration is covered elsewhere at this meeting.
EPICS-based control systems include operator interfaces which consist of multi-screen UNIX workstations using a graphical user interface based on the X-windows Motif model. The workstations support external software interfaces including: Wingz, Mathematica, accelerator modeling codes, C++, C, and any application software a physicist may write. Auxiliary displays using X-window terminals and projection and large screen displays in the main control room and seminar rooms are easily implemented using commercial equipment.
EPICS provides a number of tools for creating a control system. This minimizes the need for custom coding and helps ensure uniform operator interfaces.
An arbitrary number of IOCs and operator interfaces (OPIs) can be supported, and as long as the network is not saturated, no single bottleneck is present. If a single IOC becomes saturated, its functions can be spread over several IOCs. Rather than running all applications on a single host, the applications can be spread over many OPIs.
EPICS consists of a set of core software and a set of optional components. The core software components are channel access-client and server software, database scanners-monitors, and a database configuration tool. All other software components are optional.
Channel access is the mechanism that provides network-transparent access to IOC databases and is based on a client-server model. Each IOC provides a channel access server which is willing to establish communication with an arbitrary number of clients. Channel access client services are available on both OPIs and IOCs. A client can communicate with an arbitrary number of servers.
IOC core software consists of a memory-resident database together with various memory-resident structures describing the contents of the database. EPICS supports a large and extensible set of record types, e.g. ai (analog input), ao (analog output), etc., each of which has a fixed set of fields. Some fields are common to all record types and others are specific to particular record types. Every record has a record name and every field has a field name. The first field of every database record holds the record name which must be unique across all IOCs attached to the same TCP/IP subnet. A number of data structures are provided so that the database can be accessed efficiently. Because they access the database via database access routines, most software components do not need to be aware of these structures.
With the exception of record and device support, all access to the database is via the channel or database access routines. Database scanning is the mechanism for deciding when to process a record. Four types of scanning are possible: periodic, event, I/O event, and passive.
A request can be made to process a record periodically; a number of time intervals are supported. Event scanning is based on the posting of an event by any IOC software component. The I/O event scanning system processes records based on external interrupts. An IOC device driver interrupt routine must be available to accept the external interrupts.
As originally developed, EPICS had several tools which were ported directly from the original GTACS and could be used to develop control systems. These included a display manager/editor, an archiver, and a sequencer.
The archiver tool allows the operator to save data for later analysis. This tool has two components: the storage module and the retrieval module. The AR storage module acquires real-time data and stores the data in disk files. ARR, the archive retrieval utility, retrieves data from AR files, displays the data in textual or graphical form, and exports the data for use by other programs.
The archiver runs on the UNIX workstation and uses channel access to access parameters of interest on the network. Many instances of the archiver can be run on the control network.
The display manager provides an interface between the operator and the control system. The display manager is capable of monitoring or modifying any field in any database distributed throughout the network via the channel access communication bus.
The screens for display are configured using a graphic display editor (EDD). EDD can be used through text input or interactively. The displays are automatically connected to the operational parameters by name. No knowledge of the I/O location is needed at display configuration. Predefined display elements are included for monitoring and modifying any parameter on the network, producing display hierarchies, strip charting, and plotting synchronous events.
The display editor and display manager run under UNIX and the X-window system.
The complete sequencer consists of two components: the state notation language (SNL) and the run-time sequencer (SEQ). The SNL provides a simple tool for programming sequential operations based on the state transition diagram concept without the complexity that task scheduling, event handling, and input/output programming involve.
Early in the EPICS development APS identified the need for tools which either were not part of the original GTACS system or which the APS staff felt had to be substantially changed to provide the functionality required to development the APS system.
The central feature of an IOC is a memory-resident database together with various memory resident structures describing the contents of the database. EPICS supports a large and extensible set of record types, e.g. ai (analog input), ao (analog output), etc. This tool allows the system designer to configure the database as required. This is a text-based tool. This tool was built on the ideas of the original GTACS DCT but the changes between GTACS and EPICS required recoding the complete tool.
The alarm manager is used to create fault trees for alarm presentation to the operator. The warnings and alarms are configured in the database. The alarm manager monitors and groups these alarm conditions into a fault tree for presentation to the operator. The alarm manager provides a useful tool for steady state operation, allowing the operator to react to excursions from normal operations.
The alarm manager runs on the UNIX workstation, and its interface to the control system is via channel access. The alarm manager can be run on many workstations where each instance can run any alarm configuration.
Alarm configureation files are currently created using a text editor. The format of the text files is predefined. The configuration files are converted to a binary file at initialization. The system engineer who sets up the alarm configuration can put in guidance for the operator. An MEDM screen can be invoked from the alarm manager to aid the operator in handling the exception.
The Motif-based Display Editor/Manager Tool (MEDM) allows staff to easily create, configure, and modify displays and then activate and connect displays to actual hardware. The display editor portion of MEDM provides process connections via an interactive display editor/control. Editing/drawing commands are used to build displays with little or no custom programming required to build applications. Process connections to the data are entered as process variable name and field name; no other knowledge of the hardware configuration is required. This tool allows the designer to modify any aspect of the display element, as well as copy, align, and group display elements. Standard drawing tools can be used to create backgrounds which can then be imported into MEDM via GIF files. This tool has been developed using the Motif toolkit. It is a superset of the DM/EDD display manager developed at LANL. DM/EDD was developed at the x-lib level. This was an area where ANL and LANL disagreed about how to provide a tool and thus each lab provided its own solution.
BURT addresses the problem of backing up and restoring sets of values in a dynamic database. Backing up and restoring are simply read and write operations with respect to the database. Although the values used during restoration are typically the product of a previous back up, reading and writing are inherently independent.
The primary means of input to BURT is via ASCII files. These files define the set of names to be read. Snapshot files are the primary output of BURT. These ASCII files start with a header section followed by the data values that were successfully read.
BURT allows the user to check that certain conditions are true before writing its values. The user does this by specifying predicates in an ASCII file called a dependency file.
As an implementation, BURT is limited to reading and writing the first n values of a field. For example, BURT is not capable of reading only the second value from a field that contains more than one value.
GDCT allows the user to build EPICS databases and visualize links between records and process variables.
Two windows appear on the screen when the tool is started; one is the tool palette, and the other is the main drawing area with menu bar. The tool palette contains items that act on the drawing area.
Short form reports from DCT can be converted to the new ".db" format and then imported into a GDCT session. The tool does not automatically render an existing DCT-generated database.
The knob manager (KM) is a tool which enables the user to use the SUNDIALS knob box to adjust the settings of process variables. KM allows dynamic knob assignments from an X-windows-compliant screen, user-defined gain for each individual knob, graphical displays for the operating range and status of each process variable, the ability to backup and restore one or multiple process variables, and the ability to save current process variable value to a file and recall these values from that file in the future.
The X-window monitoring of channel access (XMCA) program is a Motif application which displays or monitors single or multiple channels of any IOC. The primary functions of the XMCA are to provide flexible loading of any database file or IOC device file based on user selection, to allow monitoring of single or multiple channels with the Motif window display system, to easily display and generate an information report of the loaded IOC, to provide a convenient IOC debug tool on the workstation, and to provide a simple backup and restore feature for the monitored channel.
The ability to quickly change the steps in a sequence has led to the development of a UNIX based sequencer. The purpose of this tool is to make the same sequencer available on the UNIX and VxWorks platforms, and to integrate and automate all the multi-platform steps formally required to build a sequence. This allows a user to easily create, modify, compile, execute, and test the state program using a menu-driven system. XMSEQ is a menu-driven X-Window Motif application.
Third-party software now interfaced to EPICS via channel access calls include Wingz, Nodal, DevTest, Mathematica, and an interpretative version of C and C++. An automatic cavity conditioning system has been developed using a Mathamatica routine running under EPICS.
A. Channel Access Security
Channel access is being modified to restrict access to the IOC database parameters from all channel access clients except those specifically authorized to monitor or control that parameter. Authorized control or modification of a particular parameter will be dependent on several factors: who is making the request; what type of access is requested, e.g., read or write; where the request is from, e.g., control room, office building, or off site; and when is the request being made, e.g., stored beam mode, storage ring orbit studies.
Consideration of the above criteria to determine access to the IOC database will be dynamically alterable by some appropriate administrative procedure. Ultimately, APS Operations will have overall control of the who, what, where, and when of IOC access. This modification is expected to be released in January of 1994.
The alarm configuration tool will be used for the design and construction of a hierarchical alarm configuration structure to be input to the alarm handler. The alarm handler tool is used by operators in the detection and handling of alarm states. As part of this project the existing user interface for the alarm handler will be changed to ensure consistency in the EPICS alarm tools.
The objective of development effort on an alarm configuration tool is to provide a rapid and efficient way for engineers and physicists, who may or may not have computer expertise, to interactively specify, build, and view an alarm configuration structure and save it into a file in the format required by the alarm handler. This file can then be used by the alarm handler to monitor database alarms.
The purpose of adding a device access layer is to change the manner in which the equipment is referenced. The original EPICS has an engineering flavor. Process variables are used for all data references. The process variables are mapped one-to-one with device I/O points. The machine physicists and system engineers want to refer to devices, not a group of I/O points, e.g., a quadrupole magnet and not the five to ten process variables that may be required to fully characterize this magnet.
The device access layer considers a device to be an object which responds to a set of messages. The semantics of devices has been designed to correspond as closely as possible to object orientated programming methods.
The first version of this library is now available. Other tools such as MEDM and the KM will be modified to have knowledge of devices.
As previously mentioned, we have examined and ported several scripting languages to EPICS. These include Nodal from CERN, DevTest from NSLS, Mathematica, Wingz, and Object Center (C++). The APS physicists were dissatisfied with aspects of all these solutions and asked for a simple, interactive language without the baggage of a commercial package or the remnants of solutions from other accelerators. To this end two APS physicists have developed an interactive language called GUS [1] (General-purpose data acquisition UNIX Shell). This language has been used extensively at the APS magnet measuring facility running on IBM-PCs and has been ported to UNIX. The APS Control Group is adding an EPICS device access layer and facilities to use MPL, a graphics and file manipulation program developed by a third APS physicist.
APS has selected to use World Wide Web (WWW) technology originally developed at CERN and the NCSA MOSAIC web browsers to place all the control system documentation on-line. This technology allows the user to easily access networked information. NCSA Mosaic is designed to provide transparent, seamless access to nearly all of these information sources and services. In addition it gives the user a mechanism to retrieve and display a wide variety of data types. These types include text, images, movies, animations, sound and scientific data. To view the APS WWW server Readers are welcome to browse the APS server at http://www.aps.anl.gov/welcome.html.
Many accelerator control systems have a device display page where each selected device and certain parameters associated with the device are displayed in a spread sheet like fashion. This tool is being added to the EPICS toolbox at the request of the APS physics group. The display will have the ability to fire off shell scripts which may be used to call up more detailed MEDM displays.
The communication network will be a combination of FDDI, Ethernet over fiber, and Ethernet over copper. Future plans call for the introduction of 100-MB Ethernet and ATM when these protocols have been standardized. The cable plant will be installed using fiber with a hub and spoke topology. All fibers will terminate in the main control room. This technique will allow APS to tailor network bandwidth as required by the application running in a particular IOC.
Subnets can be driven from the IOC. GPIB, RS-232, and Bitbus are presently supported. These subnets allow various instruments and low-cost, few-point interfaces to be connected and controlled. The Bitbus can be used to make remote, multidrop connections to GPIB and other interface subsystems.
The pace of APS installations is accelerating. The main control room is scheduled to be released from construction status in December 1993. Installation of all these systems, while actively developing and improving the control system, will remain our greatest challenge. We want to thank all the members of the APS-ASD Controls Group, the APS-XFD controls people and all the other laboratories who have contributed ideas, software, and hardware to our effort.
[1] Y. Chung and K. Kim, "Development and Application of General Purpose Data Acquisition Shell (GPDAS) at Advanced Photon Source," Proc. 1991 IEEE Particle Accelerator Conf., San Francisco, CA, Vol 2, 1299, (1991).