Synchronized Time Stamp Support

Author: Jim Kowalkowski

Introduction

Purpose

New software has been added to IOC core to maintain time stamps. The new software has the ability to maintain time stamps over all IOCs on a network. The purpose of this paper is to explain how EPICS will synchronize the time stamps. In addition, this paper will explain how to configure and use the new EPICS time stamp support software.

Definitions

Time Stamp: Two long words representing the time in seconds/nanoseconds past an epoch. The first long word represents the seconds past the epoch, the second long word represents the nanoseconds within the second. The EPICS epoch is currently January 1, 1990. A commonly used Unix epoch in January 1,1900 or in vxWorks case, January 1,1970.

Event System: A hardware based subsystem for delivering events to all IOCs on a network. Typical events are the "tick" event and the "reset counter" event.

Event System Synchronized Time: Time stamps maintained using an event system. The clock pulses which increment the time stamps are provided by a single source, all IOCs increment time stamps using the single source pulse.

Soft Time: Each IOC increments the time stamps using it's own internal clock.

Master Timing IOC: The IOC which knows the actual time and is the source of the actual time to all those who inquire.

Slave IOC: An IOC which relies on a master IOC to provide the actual time. This IOC will keep it's time stamp in sync with a master.

Soft Slave: An IOC which uses Soft Time and synchronizes it's time stamp with a master.

Event Slave: An IOC which uses Event System Synchronized Time to maintain it's time stamp. This type of IOC uses the master to verify that it's time stamp is correct.

Soft Master: A Master Timing IOC that maintains it's time stamp using a private clock.

Event Master: An IOC which uses Event System Synchronized Time and is a Master Timing IOC. In addition, this IOC is the source of the clock pulses.

Overview

Time stamps are maintained across IOCs using UDP with a master/slave relationship. A master timing IOC is responsible for knowing the actual time. A slave IOC uses the master to verify it's own time stamps. Time stamps are maintained in two fashions: using an event system or using IOC tick counters. Both differ radically and require some explanation.

Event System Support

The time stamp software has special requirements in order to provide event system synchronized time. The time stamp software assumes that hardware is present which has finite counters for counting pulses or ticks delivered to it from some master tick generator. The software further assumes that every so other, before the finite counters overflow, a signal or event will occur to reset the counters to zero. The software requires access to these counters at any time and notification when the "counter reset" arrives. The combination of hardware tick counters and reset event notification gives the software the ability to maintain time. If the event system supports other events, it is required that the software be notified of the occurrence of each event. The event time will be recorded in a table which holds one time stamp for each possible event so that a user can inquire as to when the event occurred.

Soft Time Support

If no event system is present, the time stamp software can operate in a software timing mode. In this mode, no events are available, only the current time can be retrieved. An IOC using soft time will increment a time stamp using the system clock (usually updated at 60Hz). The master/slave relationship allows a slave to periodically ask a master for the correct time. At this point the slave's time stamp can be adjusted to match the masters.

Role of a Master Timing IOC

A master timing IOC has difference responsibilities depending on if it is an event master or a soft master. An event master will be an event generator (create "tick" and "counter reset" events). When the event master detects a "counter reset", it broadcasts the time the event occurred to all event slaves on the network. The slaves can use the time stamp data to verify their own time. A soft timing master does not have any events, so no broadcasting is done. Both soft and event masters have the ability to be queried at any time by slaves for the current time.

Role of a Slave Timing IOC

A slave timing IOC also has difference responsibilities depending on if it is an event slave or a soft slave. An event slave contains hardware to manage "tick" and "counter reset" events. The time stamp support software uses this information to maintain time. An event slave will listen for broadcasts from the event master timing IOC and use the information to verify it's time. A soft slave periodically queries a master timing IOC to get the current time.

Design Specifications Summary

Event System Synchronized Time

All IOCs will have identical event times. In addition, all IOCs will maintain the same current time.

The minimum events which must be supported is two. One must be used as a "tick" event and one must be used as a "reset hardware tick counter" event. The second will be used to update the time stamp representing the current time.

An optional event can be used as a "heartbeat" event. This event can be used to signal errors.

Soft Time

All IOCs of this type will maintain time stamps which are within two clocks ticks or 1/30th of a second of a master. The master may be a designated IOC or the Unix boot server. A Unix boot server master or the server pointed to by the EPICS environment variable EPICS_TS_NTP_INET must have NTP available for polling.

User Level Interface

General Use Functions

Three functions exist in the synchronous time stamp support software for the user to retrieve time stamps:

To retrieve the time stamp which represents the time that an event occurred, use TSgetTimeStamp(). TScurrentTimeStamp() can be used to retrieve the time stamp which represents the time of day. TsLocalTime() is the function for returning the current time stamp in the old time stamp driver, the routine can still be used to retrieve this information. The function TSgetFirstOfYear() attempts to give the caller a time stamp representing January 1 of the current year relative to the vxWorks epoch 1970.

An EPICS enironment variable named EPICS_TS_NTP_INET exists which can be set to point to an NTP server. The default NTP server is the IOC boot server.

Test Functions

The following functions can be run from the vxWorks shell to give information about the time stamp driver.

Debugging Information

A global variable exists named TSdriverDebug. Setting this variable to a positive value in the vxWorks start up script will inform the time stamp driver to print information about what it is doing. The greater the value, the more information the driver will print. The number can be set and adjusted to any value any time while the IOC is running.

Record Support

Record support will have the ability to tie the record processing time to an event. This means that a user can specify that processing of a record is due to an event (from the event system). When the record gets processed, the time in the TIME field of the record will be the time when the event occurred. In order to support the event times from record support, two new fields will be added to dbCommon. The fields will be Time Stamp Event (TSE), and Time Stamp Event Link (TSEL). The TSE field will be the actual event time the user is interested in. The TSEL field will be a link used to populate the TSE.

To facilitate the use of the time stamp support software, a new record support function will be added:

This routine uses TSgetTimeStamp(prec->tse, &prec->time) to set the processing time of the record. The new recGblGetTimeStamp() will replace the existing call to tsLocalTime(). If the TSE field is zero (the default), then TSgetTimeStamp() will report the current time. It is important to remember that if a TSE field is set, then the processing time (in field TIME) will always reflect the last time the event occurred.

Driver Configuration

The synchronous time stamp support software is configured by calling TSconfigure() from the "startup.cmd" file. The parameters to this routine are:

This routine must be run before iocInit(). The synchronous time stamp support software is initialized as part of iocInit. Running TSreport() after iocInit() will produce a report which shows the current state of the driver.

Event System interface

The event system interface consists of seven function which can be provided by an event system. The synchronous time stamp support software uses a card number of zero on all functions that require a card number. The functions are as follows:

The definition are as follows:

All of these routines are checked to exist when the time stamp support code initializes. All have some kind of default routine provided if they are not found, most of which just return an error condition. The functions which the time stamp support software registers (event and error) have the following format:

Here the Card is the event system board of interest (always zero), the EventNum is the event that occurred, and the Ticks is the number of ticks since the last board tick counter reset.

In addition to the above functions, a global variable exists on the IOC which can be used to indicate that direct time is available. Setting the variable TSdirectTimeVar to a nonzero value has the same effect as providing the ErDirectTime() function.

Creating Direct Time Support

Most of the above interface functions apply only when special event hardware is present. A much simpler configuration is when a GPS is present and time stamps are distributed using IRIG-B to all or most of the IOCs. The easiest way to implement this scenario is to define ErDirectTime() to return the value one and define ErGetTime(). The job of ErGetTime() will be to actually be to generate and return the time stamp representing the current time (from the EPICS epoch). At system initialization, the actual time is retrieved from a Unix server through NTP or the Unix time protocol, so that year will be valid in the vxWorks time clock. At the GPS driver initialization, the vxWorks function clock_gettime() can be used to calculate the year. Record support will internally eventually call ErGetTime() each time the record is processed. The GPS driver should be included as part of drvSup.ascii, so it initialized in the proper order.

Operation

An IOC can be configured to run in one of four ways: synchronous-master, soft-master, synchronous-slave, soft-slave. Each mode maintains time differently. When the time stamp support code initializes, it determines the mode which it will operate in based on configuration parameters and event system function information.

Synchronous Time

Synchronous timing is determined by the presence of event system hardware. The function ErHaveReceiver() gives this information. All IOCs using synchronous time will have the exact same time stamps. The "tick" event increments all IOC's event system tick counters, the "reset tick counter" resets the counter to zero and posts the event to the time stamp support software. The time stamp support software system knows the current time by looking at the last "tick reset event" time and querying the event system with ErGetTicks() for the number of ticks which have elapsed since the last reset. Since the event handler function gets called with the "ticks since last reset", an event time is the last "tick reset event" time plus the ticks count in the call.

Master

A synchronous master is determined by the first parameter to TSconfigure() and the presence of event hardware. A master must have an event generator and an event receiver. Since the event system is configured from EPICS database records, the database must have records in it to initialize the event system. The master is responsible for providing the "tick" event and the "reset tick counter" event.

When the time stamp support software's event handler function on a master timing IOC receives a "reset tick counter" event, a time stamp message is broadcast out on the slave_port (configured by TSconfigure()). The master timing IOC is also listening on the master_port (also configured by TSconfigure()) for incoming requests for time stamp information.

At boot time, a master will set the vxWorks clock from the Unix boot server time. The time is retrieved from the boot server using NTP or the time protocol, or the server pointed to by the EPICS environment variable EPICS_TS_NTP_INET using NTP. The GPS module takes an unknown period of time to sync to the correct time, so it can not be used until it's time is valid. The event system is not up and running until record support initialization is complete, therefore the event system time stamps are not ticking until the event system initializes. At the time of the first sync, the event system is known to be up, and the "reset tick counter" or "sync" event is set to the current time (vxWorks clock).

Slave

A synchronous slave is determined by the first parameter to TSconfigure() and the presence of event hardware. This mode is automatically selected if no TSconfigure() call is used in the "startup.cmd" file. This type of slave must have an event receiver. The EPICS database must be configured to initialize the event system.

When a synchronous slave configuration is determined, the IOC broadcasts a request for master on the master_port port. If a master is found, then all current time, sync rate, and clock are extracted from the master's response (the sync rate and clock rate from the TSconfigure() are overwritten to match the master's configuration). If a master is not found, the slave IOC goes into a polling mode to try to find a master every two minutes. While a slave has no master, The IOC's clock is initialized from the Unix boot server and TSgetTimeStamp() returns the vxWorks time clock value.

The slave experiences the same problems as the master upon boot.

Soft Time

Soft time IOCs use the 60 hertz clock available from vxWorks to maintain a time stamp. The IOC will increment a time stamp at a 60 hertz rate. Soft time is determined by the absence of event system hardware. All soft timing IOC's will not have the same time stamps.

Master

A soft master is determined by the first parameter of TSconfigure() and the absence of event hardware. Upon boot, the master soft timing IOC retrieves the current time from the Unix boot server. The IOC sets up a soft time-stamp counter using a one tick watch dog and sets the vxWorks clock. From this point on, the master runs using the soft time-stamp counter. The master listens on the master_port port for requests for the master's current time.

Slave

A soft slave is determined by the first parameter of TSconfigure() and the absence of the event hardware. This mode is automatically selected if no TSconfigure() is present in the "startup.cmd" file. The basic operation of a soft slave is to synchronize the time stamp with a master when possible or to the Unix boot server if no master is available. Upon initialization the time stamp support code determines if a master is present on the network, and if NTP support is available from the Unix boot server or the server pointed to by the EPICS environment variable EPICS_TS_NTP_INET. The slave will request time stamps from a master or unix server at sync_rate_in_seconds rate from TSconfigure(). If the time stamp on the slave is found to be off, the clock will be incrementally adjusted so that by the next sync, the clock will be corrected to match the master's time stamp. A soft slave will automatically switch between a master and the unix boot server depending on if the master is available or not. In order to sync with the unix boot server, NTP must be available there for query only.

Home page for Jim Kowalkowski .

jbk@aps.anl.gov (Jim Kowalkowski)
updated 8/10/95