Experimental Physics and
Industrial Control System

Marty Kraimer <[email protected]> · Mon, 08 Dec 2003 09:35:33 -0600

This is a long answer to David's message so let me give the summary first.
SUMMARY:

Currently EPICS base 3.14 only supports thread priorities on vxWorks, RTEMS, and
WIN32. What David is doing is certainly of interest.
END OF SUMMARY
First a few comments about the existing implementation of threads and mutexes in
<base3_14>/src/libCom/osi/os/posix.
The existing implementation works on solaris (at least 6,7,8,9), linux (at least
several releases of redhat), and darwin.
HOWEVER:
Threads:
The following statements appear:
/* Until these can be demonstrated to work leave them undefined*/
#undef _POSIX_THREAD_ATTR_STACKSIZE
#undef _POSIX_THREAD_PRIORITY_SCHEDULING
The big one is _POSIX_THREAD_PRIORITY_SCHEDULING. This means that priority
scheduling is NOT being used on solaris, linux, or darwin.
Mutex:
The following statements appear
/* Until these can be demonstrated to work leave them undefined*/
/* On solaris 8 _POSIX_THREAD_PRIO_INHERIT fails*/
#undef _POSIX_THREAD_PROCESS_SHARED
#undef _POSIX_THREAD_PRIO_INHERIT
The big one is _POSIX_THREAD_PRIO_INHERIT. But, of course, it will only be of
use when _POSIX_THREAD_PRIORITY_SCHEDULING is supported.
Also osdThread has THREE!! implementations of a mutex:
1) uses spin locks. I am sending a separate message about this to Jeff and David
because I am confused about the implications.

2) Uses pthread_mutex_xxx IF the pthreads implemention support recursive mutexes.

3) Uses a combination of a pthread_mutex and a pthread_cond if the pthreads
implementation does not gurantee recursive mutexes.
It is certainly desirable to support at least _POSIX_THREAD_PRIORITY_SCHEDULING
and _POSIX_THREAD_PRIO_INHERIT wherever possible. Who should we do this? I see
two major problems: 1) Priority Scheduling, and 2) different platforms implement
different subsets of pthreads.
Priority Scheduling, at least implementing real-time priorities, gets us into
the problem of privileges. Some kind of privilege (often root) is required. How
will an EPICS IOC do this? Running everything as root is NOT a solution.
Different Platforms: Not only different platforms but different releases. This
is a MAJOR problem. I have seen some strange behavior on redhat 8 that went away
with redhat 9. I had big problems getting things to work on both solaris 6 and
solaris 8. If we turn on _POSIX_THREAD_PRIORITY_SCHEDULING for all platforms, I
bet that most (All?) will fail.
What should we do?
For Priority Scheduling work must be done to develop a good environment. Eric
Norum's message could be a start.
With regards to threads and mutexes I see two approaches:
1) Keep all pthreads code in src/libCom/osi/os/posix and use #ifdefs to take
care of differences between platforms and releases of platforms.
2) Keep src/libCom/osi/os/posix but leave osdThread and osdMutex "dumbed down"
just like they are now and provide separate implementations for platforms that
are carefully tested.
I say do 2). I suspect that 1) will lead to a set of code that will be extremely
difficult to understand and maintain.
Even if we do 2) the problems of releases still arises.
Now I have a question for David.
When the 3.6 Linux kernel is available won't it provide most of what redhawk
linux provides?
Now I will answer some specifics:

David Kelly wrote:

Hi all,
I'm porting the EPICS R3.13.4 ioc base code to RedHawk Linux, and am seeking
some advice re several questions that I have.
(But first, some background info may be helpful.... I work for Concurrent
Computer Corp, who have a real-time version of Linux for x86 platforms,
called RedHawk Linux. This is not a real-time kernel which runs underneath
standard Linux, which is how RTLinux and L4-Linux work. Rather, it is an
open source enhanced version of the Linux kernel which includes all the
standard Linux stuff plus things like: true real-time priority scheduling,
fully pre-emptible kernel, a plethora of fast interprocess synchronization
and communication mechanisms, high resolution clocks/timers, extended memory
mapping capabilities, and the ability to shield processors from interrupts
and dedicate processes to specific processors in order to guarantee
deterministic system response, etc. It is currently based on the 2.4.21
kernel from kernel.org. It is installed by first installing Red Hat 8.0 on
the target system, and then installing the RedHawk kernel and real-time
libraries, etc. It is Red Hat 8.0 compatible. A propaganda page is here
http://www.ccur.com/isd_solutions_redhawklinux.asp if anyone wants more
info.)
The work I've done so far includes:
1) Create a "redhawk-x86" build environment.
2) Moved a few elements of the "posix" source files to the new "redhawk"

branch of the source tree and modify them to better take advantage of some

RedHawk features. The work done does not affect a lot of code, and it

remains largely based on the posix base. Most of the work is as follows,

though there some additional bits and pieces:

- osdMutex: now uses RedHawk fast mutex mechanisms.

- osdThread: has now enabled the existing

_POSIX_THREAD_PRIORITY_SCHEDULING code. This was not sufficient to enable

priority scheduling under RedHawk, so a small amount of extra code was

required (e.g. specifically enabling "SCHED_RR" [or SCHED-FIFO] on a

per-thread basis).

- osdTime: Now uses CLOCK_REALTIME_HR for highest possible resolution. 

From what I answered above, I agree with the idea of creating a new
subdirectory libCom/osi/os/redhawk
Eric and I also mentioned the priviledge problem.
3) Compile the ioc base with all the changes and run the "epicsXXXXTest"
programs.
I would appreciate advice in two areas: some specific source code questions,
and then an initial much higher level question. (My biggest problem at the
moment is that I'm "looking at the wood, not the trees" - that is, I'm
hacking bits of low-level code without having much of an understanding of
how epics works as a whole. I have, however, read through the EPICS I/O
Controller Application Developer's Guide, some other EPICS training-type
docs from the web, and some whitepapers/technical reports which give an
overview of porting EPICS. Overall, these have been very helpful.....)
Firstly, I'd appreciate some help with code-specific queries:
a) The posix/osdThread.c routines uses "pthread_mutex_XXXX()" functions for
synchronization, rather than the platform specific mutexes in osdMutex.c. Is
there any reason for this ? (I've changed osdThread.c to use the mutexes in
osdMutex.c and everything seems OK so far...)

I answered this above. I agree with your approach.
b) The posix/osdTime.cpp file appears to contain current references to
"gmtime_r()" and "localtime_r()" functions, which I gather are specific to
VxWorks. These do not appear to be resolved anywhere in a build under Linux
(or RedHawk). Is this correct, or am I missing something ?

These are defined in the Single Unix Specification Version 3 (The successor to
POSIX standards). See

http://www.unix.org/single_unix_specification/
c) I don't yet understand the way the thread priorities are calculated in
posix/osdThread.c, when _POSIX_THREAD_PRIORITY_SCHEDULING is enabled. Can
someone please explain to me the effect that the getOssPriorityValue()
function is intended to have ?
getOssPriorityValue()
Given a epicsThread priority return a OS specific priority for it.
epicsThread priorities are from 0 to 99 with a higher number meaning higher
priority.
On vxWorks priorities are from 0 to 255 with a lower number meaning higher
priority. For the vxWorks implemenation we decided to map epics priorities
0,...,99 to vxWorks priorities 199,...100.

d) Under RedHawk, if a process desires to run with a real-time priority
(i.e. use a real-time scheduling policy of SCHED_RR or SCHED_FIFO, where the
real-time priorities range from 99 down to 1 and are all therefore higher
than the standard Linux SCHED_OTHER timesharing priority of zero), then the
process must be owned by, or run in the context of, the root user. I could
do this by logging on as root and MAKEing the code as root. However, I
prefer to MAKE the project in the context of my normal user login, which
means that I then need to "chown root XXXX" and "chmod a+s XXXX" all the
created binary executables so that they run as the root user. I'd like to be
able to automate the "chown" and "chmod" as part of the build (MAKE)
process, but I haven't yet figured out how to modify the build environment
to do this. Can anyone suggest how to do this, or at least where I should be
looking ?

I suggest that you use SCHED_FIFO. This is what we do for vxWorks.
The problem of root was discussed above and by Eric Norum's message.
e) What other areas of the code should I be looking at, in order to improve
the real-time performance (in a Linux-type environment) ?

Everywhere :-)
Now for the "trees"-type question....
i) I'm doing this work on my laptop computer, with some further testing on a
multi-processor system in our office. In this restricted environment, what's
the simplest way to test that the resultant EPICS "application" runs
correctly ? In the same vein, what's the simplest way to run a
performance/response-time test of the EPICS/ioc code ? (I'd like to run some
performance/response measurements under standard Linux and then again under
RedHawk. Am thinking of doing something like that described in the "Porting
EPICS to L4-Linux Based System" whitepaper, where a thread grabs the CPU
counter, kicks an I/O module into interrupting, again grabs the CPU counter,
and then measures the difference in the counter values....)

Perhaps Odagiri-San ([email protected]) could comment. He is the one who
did the whitepaper.
Thanks for any and all advice.
Kind regards,
David Kelly

Marty Kraimer

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System