Experimental Physics and
Industrial Control System

Hi all,

A year ago, Matt Rippa reached to the list[1] because he was having trouble setting the time zone at runtime on EPICS+RTEMS.

To remind you the problem: the firmware for some of our VME boards' firmware is PPCBug bootloeader, and for some others it is MOTLoad.

In the MOTLoad-based CPUs there's not much of a problem: we could just set epics-tz in MOTLoad the environment, and that will be picked-up by RTEMS when reading info from the NVRAM, defining in turn the environment variable TZ. This is not possible (or at least it is not implemented) for the PPCBug boards, meaning that we have to set the time zone in some other way. epicsEnvSet() would let us set the appropriate variables from the startup script, but at this point in the execution, there was no way to call tzset() to make RTEMS/newlib change the system-wide timezone.

Michael Davidsaver proposed a patch[2] introducing "zoneset" as an iocsh function, precisely to be able to do that (set TZ, then call tzet()).

It worked for us, but we've noticed an inconsistency: sometimes we get an unexpected behaviour, that we won't see in the next reboot.

Reading rtems_init.c:Init (the beginning of _main_ thread) I've realized that most probably this is due to a race condition. The startup sequence (simplified) goes like this:

  1) Read config from NVRAM

  2) Set the priority for _main_

  3) Init the console

  4) Start the network and initialize the remote filesystem

  5) Use the BSP supplied time of the day, if available

  6) Set TZ (if needed) and call tzset

  7) osdTimeRegister() (creates the NTPTimeSync and ClockTimeSync threads)

  8) Execute the startup script

  9) Call main()

Now, up until 7), there are only 3 threads in the system (_main_, ImsgDaemon, errlog), but this step creates two new threads to synchronize time sources.

My conclusion is that there is a race condition where the call to zoneset() (which would be the first in the startup script), does sometimes execute before the sync threads get to set anything in the system; but sometimes the sync threads manage to do some significant work before the time zone is changed to the desired one, screwing up the internal view that the EPICS/RTEMS/newlib system has of the time (we use also use Bancomm bc635 as an additional time source in some of those systems, which may, or may not, be further compounding the issue).

I'm not 100% sure of the exact effects of such a race condition, as I'm still going through the code trying to figure out all the implications.

Our fix so far has been to use instead the epicsRtemsInitPostSetBootConfigFromNVRAM hook, which is not as flexible (needs to be compiled in, where zoneset goes in the startup script), but is 100% stable.

Now... Is there any reasonable solution to this?

Regards,

Ricardo

[1] http://www.aps.anl.gov/epics/tech-talk/2016/msg00720.php

[2] http://www.aps.anl.gov/epics/tech-talk/2016/msg00733.php