Experimental Physics and
Industrial Control System

"Konrad, Martin" <[email protected]> · Tue, 3 Dec 2013 18:34:44 +0000

Hi,
I'm working on the Global Timing System for FRIB. In this context I'm 
wondering which time we should use for the whole facility. This question 
  is about two things:

1. the time we want to use (something like local time, UTC, GPS time, TAI)
2. the representation of this time in a computer/FPGA/PLC/IOC/...

We still have a few years until FRIB will be in operation so we have the 
opportunity to do things right from the beginning. I'd appreciate to 
hear your thoughts about this topic since this seems to be a critical 
design decision.

Regarding 1.:

In my eyes using daylight savings time is calling for trouble. Think 
about an archiver that has to deal with gaps and getting samples for the 
same hour twice. This is also about all the clients, scripts, and 
operators that need to keep this in mind.

Using local (winter) time might be ok for the EPICS environment itself. 
But it will be difficult to tell which of two firmware files generated 
in different time zones is newer. And we probably do want to use the 
same time for everything...

UTC solves these problems but has leap seconds. Unfortunately they are 
only known a few months in advance which makes it difficult to build 
them into computers not to mention a distributed control system.

If you want to get rid of leap seconds you can go for GPS time [1] or 
International Atomic Time (TAI, [2]). Both do not have leap seconds and 
thus slowly diverge from UTC (right now GPST = UTC + 16 s, TAI = UTC + 
35 s).

TAI and GPST seem to be ideal to me for an accelerator control system. 
They can be easily derived with high precision from GPS clocks. The 
downside is that operators have to work with different times on their 
GUI and their watch.

Regarding 2:

It seems like most of our current systems use POSIX timestamps or EPICS 
timestamps for internal representation of time. Unfortunately they both 
come with several issues:

1. 32 bit POSIX timestamps will overflow in 2038 [3] which is within the 
planned life time of FRIB. EPICS timestamps are very similar but use a 
different epoch and thus can be used until 2058.

2. POSIX as well as EPICS timestamps do not provide a representation for 
leap seconds (although the clock is running on UTC by definition). Thus 
one of the following ugly work-arounds is needed:

a) Wait for the clock to be off one second and compensate for that using 
NTP after it happened. Not acceptable since every machine corrects at 
its own speed.
b) Make the preceding second last two seconds by stopping the clock for 
one second. This is done by systems running recent NTP daemons and 
kernels. Since these systems do not have a correct time during leap 
seconds they cheat by simply providing _some_ time that they think 
_might_ make sense. There is no guarantee that you will get the same 
timestamp at the same time on multiple machines (even if they run the 
same OS!). We also have to make sure all our software can handle twice 
as much samples per second etc. Probably not acceptable.
c) Start some time earlier and stretch seconds a little ("smear" out the 
leap second). This approach is used by Google for their servers [4]. If 
implemented in the right way this should happen synchronously on all 
machines. This probably cannot be done with PLCs.

3. Leap seconds make it much more difficult to calculate the time 
elapsed between two timestamps (which is what nuclear physicists usually 
do when working with their data). We need to make sure we have correct 
leap-second data available for all software that calculates time 
differences. This seems to be practically impossible to me.

It seems to be a _really_ big task to determine if all system on our 
accelerator network handle leap seconds correctly or even the same way.

My proposal/solution:

Use TAI for the whole accelerator network including all EPICS IOCs, 
PLCs, control room PCs etc. Force operators, users, logbooks etc. to use 
TAI. Keep using local time for office PCs and data center equipment.

Use TAI64 timestamps [5] where possible. Convert to POSIX/EPICS 
timestamps where needed. The latter should work smoothly until 2038 
since TAI has no leap seconds. Control System Studio's PVManager already 
uses 64 bit timestamps internally. Since most state-of-the-art CSS 
applications use PVManager I don't see any fundamental problems on this end.

To use TAI we have to run our own PTP and NTP servers and prevent the 
latter from announcing leap seconds. We need to block access to NTP 
servers outside of the accelerator network.

There are still some open questions:

What effects do different timestamps have on network connections that 
cross the office/accelerator network boundary? Some things that come to 
my mind:

1. Encrypted connections might cause problems because system time is 
used to protect against replay attacks. Usually a time difference of one 
or two minutes should be acceptable, though.
2. Functions of GUI software sending the current system time via the 
network might result in undesired behavior if used from the office 
network. E.g. Control System Studio Archive Viewer's "-10 minutes to 
now" probably will not work as expected.

What else do we need to keep in mind? Can anyone provide some experience 
with running a control system on a time that is different from local 
time/UTC?

Thanks for reading this long one!

Martin

[1] 
http://en.wikipedia.org/w/index.php?title=Global_Positioning_System&oldid=584232968#Timekeeping
[2] 
http://en.wikipedia.org/w/index.php?title=International_Atomic_Time&oldid=581789205
[3] http://en.wikipedia.org/w/index.php?title=Unix_time&oldid=583201834
[4] http://www.aps.anl.gov/epics/tech-talk/2011/msg01579.php
[5] http://cr.yp.to/libtai/tai64.html

-- 
Martin Konrad
Control System Engineer
Facility for Rare Isotope Beams
Michigan State University
640 South Shaw Lane
East Lansing, MI 48824-1321, USA
Tel. 517-908-7253
Email: [email protected]

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System