7. APS/ASD Configuration Management Procedures

7.1. Overview

This section describes a set of procedures for managing ioc software. EVERYTHING is for the sole purpose of supporting iocs, thus it does not discuss management of high level applications. The procedures are intended for an operational facility. During commissioning activities some more relaxed procedures may be appropriate.

The Configuration Management process described here relies on a certain level of understanding and responsibility by the engineers who implement it, and may not always rigorously maintain a previous "known working" set of software online for an IOC to be rolled back to in the event of problems while making changes. It is always possible to retrieve the last version from the system backup tapes, but this may not be acceptable for some high-availability applications. Sites desiring more beaurocratic but "safer" procedures may wish to contact JLAB or SNS to find out about their approaches.

The key features of the APS Configuration Management System are:

CVS
All human created/edited files are put in the cvs repository dedicated to ioc applications
<top>
The complete set of ioc software is divided into <top> areas. Each <top> area is managed independently. Each follows its own development and release schedule.
<ioctop>
This is a <top> dedicated to a set of iocs that are booted from <ioctop>/iocBoot/iocxxx
<supporttop>
A <top> area that contains products shared across multiple <ioctops>.
Releases
Each <top> can have multiple releases.
Operations and Development areas
Releases of <top> appear in an operations tree. Operational iocs are booted from this tree. Developers can checkout any <top> in a private tree.
<supporttop> releases are not modified in the operations tree.
Because a <supporttop> is used by multiple <ioctops> it should never be modified in the operations tree. Any modifications require a new release. It may be appropriate to relax this rule during a period of commissioning.
<ioctop> releases may be modified in the operations tree.
This makes it easy to make small ongoing changes to operational systems.

Background

Before EPICS base release 3.13, APS/ASD used an Application Source Release control system called appSR. appSR uses SCCS for source file control. Since appSR was written several things that impact S/R control have changed.

With release 3.13 the method of configuring record/device/driver support changed.
Both EPICS base and extensions have switched from sccs to cvs .
base/config, i.e. definitions and rules for GNU make, has evolved so that is now possible to create makefiles that are much simpler, more powerful, and extendable.

For the above reasons we decided to redo the APS/ASD IOC Applications S/R control system. Our goals were:

SCCS will no longer be used.
All functionality performed by appSR will be done with a combination of CVS and GNU make.

A major decision was What directory structure should we use. Before the original appSR was written a lot of discussion went into this topic. There were several meetings between the Application Developers and Bob Zieman, who implemented appSR. This directory structure was used to implement the entire APS/ASD contol system software. Thus a lot of correct decisions were made. The only major problem is that it was not easy to share code, mainly record/device/driver support, across <top> areas. This resulted in sharing by copying source modules from one <top> area to another. When one developer would make changes to a source module, the other developers would often not even be aware of the changes. Thus over time the source modules evolved in different directions.

Since the overall directory layout appears to be correct, it is kept in the new system. A brief description of the APS/ASD environment may help explain why the directory layout works.

APS accelerator
- The APS accelerator complex is composed of four major subsystems: Linac, PAR (Positron Accumulator Ring), Booster Syncrotron, and Storage Ring. Each subsystem is controlled by a separate set of IOCs. Thus it is entirely appropriate to have separate <top> areas for each subsystem.
- Each subsystem can be viewed as a separate set of applications, e.g. RF, Magnet Power Supplies, Diagnostics, etc.
IOC responsibilities.
- A particular application may be spread over multiple IOCs. For example the control for storage ring applications is normally spread over many IOCs just because of the physical size of the storage ring.
- A particular IOC may contain parts of multiple applications. For example vacuum and power supplies normally share IOCs.
Application Developers.
- For the Linac, a single Application Developer has final responsibility for all controls applications, i.e. RF, vacuum, Diagnostics, etc.
- For the par, booster, and sr, Application Developers are assigned application areas. For example the same person is responsible for almost all diagnostic controls for the par, booster, and storage ring.

One other topic to discuss before describing the overall directory structure is the idealized Application Development Cycle. It consists of the following steps:

Define I/O requirements.
This involves meeting with the user, in this case the engineers who are responsible for the application in order to decide the types and number of I/O modules needed.
Assemble control hardware and software.
This includes IOCs, I/O modules, software device/driver support, etc. If EPICS software support is not already available, it has to be written and tested.
Build databases, sequence programs, etc.

In reality there is overlap between these steps. In addition as new needs arise the three steps again have to be performed. However, an application developer tends to spend a large part of his/her attention on each step before moving on to the next step.

7.2. CVS Repository for IOC Applications

The cvs tree is arranged in two parts:

$CVSROOT/
    support/
        <supporttop1>/
        <supporttop2>/
        ...
    ioc/
        <ioctop1>/
        <ioctop2>/
        ...

So at APS/ASD the repository tree is:

$CVSROOT/
    support/
        base/
        allenBradley/
        mpf/
        share/
        bitBus/
        radMon/
        switchgear/
        runcontrol/
        motorTransform/
        saverestore/
        PSCU/
        ...
    ioc/
        sr/
        mcr1/
        runcontrol/
        tune/
        s40misc/
        srbpm/
        ...
    ...

7.3. Adding a new <top> to the CVS repository

This section provides guidelines for creating a new <top> application that will be maintained in the local cvs repository, i.e. it does NOT apply to <top> applications maintained somewhere else.

Assume you have a directory tree starting at <working>/<newtop> that you want to put in the cvs repository at location $CVSROOT/<system>/<newtop>Just issue the commands:

cd <working>/<newtop>
gnumake clean uninstall
cvs import -m "Creating" <system>/<newtop> <newtop> start

where <system> is either support or ioc.

A .cvsignore file should appear in all <top> areas and also in all <top>/iocBoot/ioxxxx directories.

<top>/.cvsignore should contain:

bin
include
lib
man
dbd
templates
doc
db
javalib
html

<top>/iocBoot/iocxxx/.cvsignore should contain:

cdCommands

.cvsignore files are added to the repository just like any other file.

Documentation and Release Notes

Each <top> should provide some documentation describing the application and also release notes describing each release. The release notes should mention any dependencies on other <top> applications it uses. The recommended documentation format, at least for release notes, is HTML. These documents can be put in any directory containing a Makefile.Host. Adding the lines:

HTMLS += xxx.html
HTMLS_DIR = .

to this Makefile will result in xxx.html being installed into the directory $(INSTALL_LOCATION)/html/., where $(INSTALL_LOCATION) is normally <top> The HTMLS_DIR value can be used to target a subdirectory under $(INSTALL_LOCATION)/html , useful for longer documents with many sections and graphics files.

7.4. Operations Directory Structure

An operations tree has the structure:

<operations>/
    RX.XX.X/
        support/
            <supporttop1>/
                1-1/
                ...
            <supporttop2>/
                1-1/
                ...
            ...
        ioc/
            <ioctop1>/
                1/
                ...
            <ioctop2>/
                1/
    RY.YY.Y/
        ...

NOTES

A new RY.YY.Y directory will be needed whenever a significantly changed version of base is needed which requires that all ioc and support applications be rebuilt to use the new version. Some examples of such major changes are:
- A new version of vxWorks is being used.
- The underlying epics structures are modified.
- The set of fields in dbCommon are changed.
<supporttop>/x-y releases are created via cvs export commands because it is not permissible to modify an operational <supporttop>.
<ioctop>/x releases are created with a cvs checkout command because it is permissible to modify an operational ioc release. See below for reasons.

If only bug fixes are being applied to EPICS base then a new RX.XX.X directory is generally not needed. Instead a new support/base/x-y-site release can be created.

A portion of the actual APS/ASD operations area is:

/usr/local/iocapps/
R3.13.1/
    support/
        base/
            3-13-1-asd1/  This is actually 3.13.1 with bug fixes
        share/
            1-3/          What is left after unbundling
        allenBradley/
            1-1/
        mpf/
            1-3/
        bitBus/
            1-1/
        radMon/
            1-1/
        switchgear/
            1-1/
        runcontrol/
            1-1/
        motorTransform/
            1-1/
        saverestore/
            1-1/
        PSCU/
            1-1/
    ioc/
        sr/
            1/
        mcr1/
            1/
        runcontrol/
            1/
        tune/
            1/
        s40misc/
            1/
        srbpm/
            1/
        others as they get converted to 3.13.1

7.5. Support Management

7.5.1 Overview

A <supporttop> must follow a few simple rules in order to be used by an <ioctop>.

If it uses EPICS_BASE or another <supporttop>, it must have a config/RELEASE file.
It must install everything intended for use by an <ioctop> into a directory immediately under its <top>.

Thus it must have the following directory structure:

<top>
    config/
        RELEASE
    bin/
        <arch>/
        ...
    lib/
        <arch>/
        ...
    dbd/
    db/
    include/

NOTE: Most <supporttop>s will need only a subset of bin, lib, dbd, db and include .

At least the following two types of support areas may be used by an <ioctop>

A <supporttop> that is developed in the same cvs repository as the <ioctops>
A <supporttop> that follows the above directory structure but is maintained in a different cvs repository. Examples are epics base and epics unbundled products such as the allenBradley support. To allow local bug fixes to be made to any of these remotely obtained products, the following management procedure is used:
- The product is exported from the cvs repository where it is maintained, or otherwise copied into a local directory area.
- This area is then imported into the local cvs repository as a vendor import.
NOTE: EPICS base will follow this procedure. This makes it easy to make local bug fixes rather than having to create and use a completely new EPICS release for even minor changes.

In addition an application might use commercial products. Since these products are not under our control, each <ioctop> has to decide how to use the product. Wherever possible, however, the product location should be defined by a line in the <ioctop>/config/RELEASE file.

7.5.2. Procedures for <supporttop> Maintained in Local Repository

Summary:

Developers develop in a private area checked out from the main cvs branch.
When a new version is ready for operations, all changes are committed and a release tag is created.
Operations creates an exported version of the new version in the operational area.
If it is necessary to patch an older release, a branch is created based on that release and the patch applied to the new branch.

Developer's Cookbook

General Guidelines:

Have a standard place to do development. At APS/ASD it is recommended that each developer use the following paths, which should be created with mkdir commands:
```
<home>/iocapps/RX.XX.X/support
```
Checkout ONLY the <supporttop> applications for which you are responsible.
If cvs watch on has been set then all files will be checked out read only. If you want to edit a file, first issue the command:
```
cvs edit <filename>
```
DO NOT just issue a chmod to change the file permissions.
Try to do all development on the main cvs branch. It should seldom be necessary to create branches for <supporttops>. If it is get help unless you understand the procedures described below. This section only discusses changes on the main branch.
If a new application is being created get help unless you understand the procedures described previously.

Initial Checkout:

A private working version of <supporttop> is checked out via the commands:

cd <support>
cvs checkout -d <supporttop> support/<supporttop>
cd <supporttop>
    all development done here

Tagging a new release:

When changes have been made and it is time for operational use, all changes should be committed and a new release tag created:

cd <support>/<supporttop>
    make sure all changes have been committed
cvs tag Rx-y

Tags should take the form Rx-y, where x is a major release number for the module and y is a minor release number. For example the first release is R1-1, the next minor release R1-2, etc.

You can see all previous tags by issuing the commands:

cd <support>/<supporttop>
cvs status -v Makefile

After tagging the new release you should notify operations to install it in the operations tree.

Creating a tar file:

If you also want to create a tar file of containing the new release for use elsewhere, you should not just tar up your working directory, but instead use the following commands:

cd <somewhere>
cvs export -kv -d <supporttop>-x.y -r Rx-y support/<supporttop>
tar cf <supporttop>-x.y.tar <supporttop>-x.y
gzip <supporttop>-x.y.tar

Note that the naming conventions used for the export directory and tar file above (a hyphen `-' separates the name and release number, and the release number has a period `.' between major and minor parts) are those used for most open source software. The -kv option to cvs export instructs it to expand RCS keywords so that they will be preserved without further change if the resulting files are ever imported into a different CVS repository, thus preserving their history.

Patching Old Releases

NOTE: This should seldom be necessary.

If it becomes unavoidable to apply patches to an old release instead of upgrading to a newer version, a branch must be created in the repository to hold these changes. The following procedure demonstrates this:

cvs rtag -b -r Rx-y Bx-y support/<supporttop>
cd <somewhere>
cvs checkout -d <supporttop>/Bx-y -r Bx-y support/<supporttop>
cd <supporttop>Bx-y
    make, test and commit changes
cvs tag Rx-y-1

The cvs rtag -b can be issued from anywhere as it creates the branch tag Bx-y by working directly on the files in the repository. The checkout command specifying -r Bx-y then retrieves that new branch. All subsequent cvs commands issued in the new area will apply to the branch rather than the versions on the main trunk. The command cvs tag Rx-y-1 makes a tag for modified version on this branch. A good convention to follow is that branch releases should have the same tag as the release with -i appended, where i represents the patch release number for this branch. Thus the first patch is Rx-y-1, the second Rx-y-2 etc.

Operations Cookbook

Managing local <supporttop> modules

Installing a new release for use in the Operations Area

Operations installs a new <supporttop> by exporting the new release into the operational directory tree:

cd <operations>/RX.XX.X/support/<supporttop>
cvs -r export -d x-y -r Rx-y support/<supporttop>
cd x-y
gnumake

Note that the operations version is created using cvs export rather than cvs checkout. This means that no CVS directories appear in the operations version of a <supporttop> . This directory area should never be modified once operational <ioctops> start using it. If a modification is required a new (possibly patch) release must be made using the methods described in this document.

Installing a patch release to the operations area

Patch releases are installed in the operations tree exactly like any other release, using the commands:

cd <operations>/RX.XX.X/support/<supporttop>
cvs -r export -d x-y-i -r Rx-y-i support/<supporttop>
cd x-y-i
gnumake

7.5.3. Managing a <supporttop> from another Repository

This applies to products like EPICS base, allenBradley, mpf, etc. The basic rules are:

The first time a product is put in the local repository:
- It is imported into support/<product> giving a vendor tag reflecting the product and a release tag reflecting the release version.
- Any local modifications that are needed (e.g. changes to config, NOT bug fixes) are committed on the main branch.
Any local bug fixes are made on a new branch, NOT the main branch.
When a new release of the product from the maintainer is to be incorporated:
- It is imported again, using the same vendor tag but with a release tag reflecting the new maintainer's release version number.
- Any local changes that occurred on the main branch between the two import operations will automatically be merged into the new maintainer's release and the result placed at the head of the main branch.
- Any conflicts due to overlapping changes between the newly imported release and local modifications will be highlighted to be resolved manually.

If possible the local release tags assigned to products maintained elsewhere should identify the imported release. For example epics base would be imported with a command like:

cvs import -m "import base 3.13.1" support/base base release3-13-1

After making and committing any local configuration changes this would be given a tag of R3-13-1. If it is necessary to patch this release of base, the first patch tag would be R3-13-1-asd1.

The following examples are for mpf. Assume that the first version of mpf is version 1-1 and the second version is 2-1.

Importing for the first time

Assume that a tar file containing version 1-1 of mpf has been extracted into the directory <working>/mpf

cd <working>/mpf
cvs import -m "Import mpf version 1-1" support/mpf mpf release1-1
cd <working>
/bin/rm -rf mpf

Now checkout the new product from the main branch, make local changes, tag it, and delete the working copy.

cd <working>
cvs checkout -d mpf support/mpf
cd mpf
    make and test local configuration changes
cvs commit -m "Configuration changes for release 1-1"
cvs tag R1-1
cd <working>
/bin/rm -rf mpf

Only changes that are needed for the local environment should be made here, for example the config/RELEASE file should be modified. Bug fixes should NOT be applied here - see below for how to make bug fixes. The local release tag R1-1 matches the mpf version number, indicating that the software is identical to that of the official mpf release but with local configuration steps fully made.

Importing a new version

Assume that version 2-1 of mpf has been extracted from a tar file into <working>/mpf:

cd <working>/mpf
cvs import -m "Import mpf version 2-1" support/mpf mpf release2-1
    The import will indicate any conflicts that need resolving.
cd <working>
/bin/rm -rf mpf
cvs checkout -d mpf -j release1-1 -j release2-1 support/mpf
cd mpf
    Make changes to resolve conflicts and add any new local configuration
    changes needed.  The following commit is needed to remove files that have
    not been locally modified and which are not present in the new release.
cvs commit -m "Merged local changes with mpf 2-1"
cvs tag R2-1
cd <working>
/bin/rm -rf mpf

The cvs import command will report any conflicts that have to be merged by hand, and provides instructions on how to do this if it is necessary. In our example this was the case, and the subsequent cvs checkout command merged the changes between the two vendor releases onto the main branch.

Applying Local Patches

If it is necessary to make local patches to software that is maintained elsewhere, the changes must be made on a side branch. If the changes were made on the main branch then importing later versions of the product that had these changes incorporated would flag these up as conflicts that had to be resolved manually. The disadvantage is that local bugfixes which have not been included in an official release will have to be re-applied on another new branch, but this can be regarded as an incentive to ensuring that the official maintainer be sent copies of all bug fixes found.

Assume that it is necessary to apply a patch to version 1-1 of mpf:

cvs rtag -b -r R1-1 B1-1 support/mpf
cd <working>
cvs checkout -d mpf -r B1-1 support/mpf
cd mpf
    make any necessary changes and commit them
cd <working>/mpf
cvs tag R1-1-asd1
cd <working>

The new release tag that operations should install is R1-1-asd1.

Exporting a new release to the Operations Area

Releases are prepared for operations just like for a <supporttop> being maintained in the local repository.

7.6. IOC Management

7.6.1. Overview

The procedures for <ioctop>s are designed so that simple changes to the operational tree are easy. It is expected that there will be frequent (weekly or even daily) small changes such as adding database records, changing HOPR, LOPR values, etc. All these minor changes are made on a side branch rather than the main branch of the cvs tree. Developers check out a private copy of this branch. The operational area, from which the iocs are booted, has the same branch checked out. Changes appear in the operational area by a developer committing changes and operations performing cvs updates.

The following diagram shows how an <ioctop> evolves over time.

Evolution of an
<ioctop> over time

A new branch is needed when:

It is not possible to boot all iocs from the same directory structure, for example while converting to a new release of EPICS base.
Major changes are being made to <ioctop> and it must be possible to quickly revert to the old working system.

Assume the current operational system is

<operations>/R3.13.1/ioc/<ioctop>/1

Each <ioctop> application developer has a private area

<somewhere>/ioc/<ioctop>/1

Both of these areas have been checked out via the commands

cd <somewhere>/ioc/<ioctop>
cvs checkout -d 1 -r B1 ioc/<ioctop>
cd 1

All cvs commands issued in an <ioctop>/1 area will be made to the B1 cvs branch. Until it is time for a new release no changes are made to the main cvs branch.

Assume that a new release of epics is available, which requires a new side branch for <ioctop>. The following steps are taken by whoever is preparing it for the new epics release:

Step 1

The main branch is checked out of the repository and the B1 changes merged back into the main branch.

cd <somewhere>
cvs checkout -d <ioctop> ioc/<ioctop>
cd <ioctop>
cvs update -d -j B1
cvs commit -m "B1 merged into main branch"

This is shown in the diagram as "B1 merged into main branch".

Step 2

Changes can be made to <ioctop> and committed without affecting the B1 branch. During this step the major development work needed is done on the main cvs branch.

Step 3

When it is ready for use by other application developer's and/or operations, a new side branch B2 is created. The R2 tag also created below marks the file versions at the root of the new branch for later reference.

cd <somewhere>/<ioctop>
cvs commit -m "changes for new epics release"
cvs tag R2
cvs tag -b B2
cd <somewhere>
/bin/rm -rf <ioctop>

Step 4

The new area is checked out via the commands:

cd <somewhere>/<ioctop>
cvs checkout -d 2 -r B2 ioc/<ioctop>

Changes can be made independently on either branch B1 or B2, and iocs can be booted from either area. Once all iocs are successfully using B2, the B1 cvs branch, operations 1 area, and any 1 private areas are "abandoned", i.e. they no longer need to be maintained. The abandoned branch ends in a hollow box in the diagram.

IMPORTANT WARNING. While B1 and B2 are both active, it is the application developer and operations responsibility to make any changes to both releases. This starts from the time step 2 is performed until B1 is no longer active. The diagram identifies this period with "dual changes". CVS can help with this task, but in most cases it will probably be easier and just as quick to make and commit the same change in both areas.

The diagram shows that the same steps are followed for branch B3.

Why is minor development done on a branch?

Development is done on a branch so that when a new branch becomes the operational branch, nothing has to be done to get the operational tree on the latest release - it is already using it!

If development were done on the main cvs branch, then when new development got to the point where it was desired to make this the new operations release, a new release would have to be generated on a side branch and the official area switched to use the new release. In practice at APS this time has often only been known once the system was already operational. The only easy way to prove that the new release is really ready is to run it in an operational mode. By then however, operations staff are already saying: "It works, please leave it alone."

Versions of a branch

<ioctop> developers may want to create a tag on a branch before and after making extensive changes to the branch or before merging a branch back onto the main tree. These tags should always be of the form Rx-y where x is the branch and y is a minor release number. This if the current branch is B2 then the tags are R2-1, R2-2, .... These tags are NOT created for the purpose of creating operational releases. They are only to tag snapshots of the cvs repository at key moments.

7.6.2. Developer's Cookbook

General Guidelines:

Have a standard place to do development. Each site may establish guidelines. At APS/ASD it is recommended that each developer work in an area under their home directory as follows:
```
<home>/iocapps/RX.XX.X/ioc/
```
Checkout ONLY the <ioctop>s for which you are responsible.
If cvs watch on has been set then all files will be checked out read only. If you want to edit a file issue the command:
```
cvs edit <filename>
```
DO NOT just issue a chmod to change the file permissions.
Do all development on the correct cvs branch. If it is time to create a new cvs branch for an <ioctop> get help unless you understand the procedures described previously.
Be careful. Remember that the operations area is using the same cvs repository as you.
Whenever possible use the same config/RELEASE definitions as operations.

Check Out an `<ioctop>` branch

cd <ioc>/<ioctop>
cvs checkout -d x -r Bx ioc/<ioctop>
cd x
    all development is done here

where x is the branch desired. For example if B1 is the current branch than x is 1.

Committing Changes

Be Careful. Remember that when operations performs updates and builds in the operations tree the operational system will be using your modifications. During operational periods only changes really needed for operations should be committed.

cd <ioc>/<ioctop>/x
    commit ONLY the files actually changed

Now prepare instructions for what operations should do. The instructions should specify cvs updates and makes at the lowest level possible. Do NOT just give instructions to go to <ioctop>/x and update and rebuild.

7.6.3. Operations Cookbook

Ask for detailed instructions from developer's about where to perform cvs updates and makes.

7.7. Additional Guidelines for Operations

Backup each <ioctop>

Because <ioctop>s can be modified, it is possible that a developer could perform cvs commits that will break an operational system when a cvs update is done in the operations area. In order to prevent a major problem, procedures should be in place to backup all <ioctop>s on a regular schedule when the facility is on an operational period. Procedures should also be in place to restore a particular <ioctop> from the backup.

Application developers must be aware that only minor changes should be made during operational periods. When a developer makes a change commits should be made only to the necessary files rather than commits at the top level. He/she should instruct operations about what should be updated and rebuilt, i.e. update just the files necessary and only build the components affected by the updates.

Updates in <ioctop> areas

Because application developers can accidentally commit changes that would break the operational system care should be taken when performing cvs updates on an <ioctop> in the operations area. A good idea is to do the following before any updates:

Wherever you plan to issue a cvs update first issue the command

cvs -n update

which will list the files that are liable to be changed. Then on any files you want to update issue the command

cvs diff -r Bx <filename>

where Bx is the current branch tag, to see the modifications that have been committed. Only if you are satisfied with these changes should you perform the actual update.

$CVSROOT/CVSROOT/cvsignore

This file should contain

O.*

which makes cvs ignore all O.* files .

Contents Previous Next