Experimental Physics and
Industrial Control System

David Kelly <[email protected]> · Wed, 10 Dec 2003 22:34:28 +1100

Thanks Marty.

I've read all the follow-up correspondence, especially those from yourself,
Jeff Hill, Andrew Johnson, and Eric Norum. If I may reply to some of the
points raised:

1) I still intend to create a Redhawk-specific version of osdMutex and
osdThread, etc. This seems to be the most easily understandable and most
maintainable solution...though I'm open to better ideas. I intend that the
Redhawk-specific version of osdMutex will only contain one implementation of
a mutex.

2) Having read the discussion about mutexes and spinlocks, etc, I'm
convinced that the osdMutex code that I've built already, needs to be
revised. At present, the code is based on "true" spinlocks (i.e. initially,
an atomic memory operation instruction, followed by spinning using a
"standard" memory read operation and then retrying the atomic lock when the
"read" indicates that the lock is available), although it is cognisant of
whether it is running on a system which has 1 CPU active, or >1 CPU active
(including logical CPUs enabled by Hyperthreading). The present code only
spins if it's running on a multi-CPU system, otherwise it calls
"sched_yield()" to give up the CPU as after it tries once, and fails, to
acquire the lock. At present I do nothing special to cope with the Priority
Inheritance problem - I confess that I don't know enough about the EPICS
code to understand if Priority Inheritance is really required, or if the
code can never exhibit such problems. However, having said that, I suspect
that the code should somehow be made to cater for PI. (A forthcoming release
of Redhawk will provide kernel support for PI on mutexes, so in that case I
could spin for a while and then use a kernel mutex - but I don't want to
have to wait for it...) I'll let you know what I wind up building.

3) Re the difference between Redhawk and the new Linux 2.6 (presumably the
"3.6" in your email was a typo) kernel...  My understanding (not gospel) is
that the following Redhawk features will come as standard to all Linux
systems in the 2.6 kernel:
  - kernel preemption
  - low-latency kernel patches (though these are only a subset of what's in
Redhawk)
  - the real-time scheduler
  - Posix timers (though only "low" resolution, not close to Redhawk's high
res timers)

This means that (at least) the following existing Redhawk features will not
be present in the standard Linux 2.6 kernel:

- rescheduling variables
- fast block/wake services
- usermap and /proc enhancements
- processor shielding and affinity
- frequency-based scheduler

Then there are new features planned for future releases of Redhawk and, as
mentioned, one of these is support for Priority Inheritance.

4) The "gmtime_r()" and "localtime_r()" functions are indeed present in
Redhawk - thanks to all for pointing out where these live.

5) I'm not sure what to do about the "must run as root" problem. I can't use
a wrapper as Eric suggested, because (for example) I can't set the
scheduling policy and priority for all threads once - at program
commencement - because this must be done as each thread is created or after
a thread is created. I'm not sure how to handle this problem - I *might*
have to really run stuff as root. I know Redhawk supports Pluggable
Authentication Modules (PAMs), but I've never looked at them in detail, so
I'm not sure if they provide the functionality required. Will have to take a
look... As always, other suggestions are welcome...

Thanks for everyone's help and advice - I really appreciate it. 

Kind regards,

David

-----Original Message-----
From: Marty Kraimer [mailto:[email protected]]
Sent: Tuesday, 9 December 2003 2:36 AM
To: David Kelly
Cc: [email protected]
Subject: Re: Novice EPICS questions re yet another port

This is a long answer to David's message so let me give the summary first.

SUMMARY:
Currently EPICS base 3.14 only supports thread priorities on vxWorks, RTEMS,
and 
WIN32. What David is doing is certainly of interest.

END OF SUMMARY

First a few comments about the existing implementation of threads and
mutexes in 
<base3_14>/src/libCom/osi/os/posix.

The existing implementation works on solaris (at least 6,7,8,9), linux (at
least 
several releases of redhat), and darwin.

HOWEVER:

Threads:

The following statements appear:

/* Until these can be demonstrated to work leave them undefined*/
#undef _POSIX_THREAD_ATTR_STACKSIZE
#undef _POSIX_THREAD_PRIORITY_SCHEDULING

The big one is _POSIX_THREAD_PRIORITY_SCHEDULING. This means that priority 
scheduling is NOT being used on solaris, linux, or darwin.

Mutex:

The following statements appear
/* Until these can be demonstrated to work leave them undefined*/
/* On solaris 8 _POSIX_THREAD_PRIO_INHERIT fails*/
#undef _POSIX_THREAD_PROCESS_SHARED
#undef _POSIX_THREAD_PRIO_INHERIT

The big one is _POSIX_THREAD_PRIO_INHERIT. But, of course, it will only be
of 
use when _POSIX_THREAD_PRIORITY_SCHEDULING is supported.

Also osdThread has THREE!! implementations of a mutex:

1) uses spin locks. I am sending a separate message about this to Jeff and
David 
because I am confused about the implications.
2) Uses pthread_mutex_xxx IF the pthreads implemention support recursive
mutexes.
3) Uses a combination of a pthread_mutex and a pthread_cond if the pthreads 
implementation does not gurantee recursive mutexes.

It is certainly desirable to support at least
_POSIX_THREAD_PRIORITY_SCHEDULING 
and _POSIX_THREAD_PRIO_INHERIT wherever possible. Who should we do this? I
see 
two major problems: 1) Priority Scheduling, and 2) different platforms
implement 
different subsets of pthreads.

Priority Scheduling, at least implementing real-time priorities, gets us
into 
the problem of privileges. Some kind of privilege (often root) is required.
How 
will an EPICS IOC do this? Running everything as root is NOT a solution.

Different Platforms: Not only different platforms but different releases.
This 
is a MAJOR problem. I have seen some strange behavior on redhat 8 that went
away 
with redhat 9. I had big problems getting things to work on both solaris 6
and 
solaris 8. If we turn on _POSIX_THREAD_PRIORITY_SCHEDULING for all
platforms, I 
bet that most (All?) will fail.

What should we do?

For Priority Scheduling work must be done to develop a good environment.
Eric 
Norum's message could be a start.

With regards to threads and mutexes I see two approaches:

1) Keep all pthreads code in src/libCom/osi/os/posix and use #ifdefs to take

care of differences between platforms and releases of platforms.

2) Keep src/libCom/osi/os/posix but leave osdThread and osdMutex "dumbed
down" 
just like they are now and provide separate implementations for platforms
that 
are carefully tested.

I say do 2). I suspect that 1) will lead to a set of code that will be
extremely 
difficult to understand and maintain.

Even if we do 2) the problems of releases still arises.

Now I have a question for David.

When the 3.6 Linux kernel is available won't it provide most of what redhawk

linux provides?

Now I will answer some specifics:

David Kelly wrote:
> Hi all,
> 
> I'm porting the EPICS R3.13.4 ioc base code to RedHawk Linux, and am
seeking
> some advice re several questions that I have.
> 
> (But first, some background info may be helpful.... I work for Concurrent
> Computer Corp, who have a real-time version of Linux for x86 platforms,
> called RedHawk Linux. This is not a real-time kernel which runs underneath
> standard Linux, which is how RTLinux and L4-Linux work. Rather, it is an
> open source enhanced version of the Linux kernel which includes all the
> standard Linux stuff plus things like: true real-time priority scheduling,
> fully pre-emptible kernel, a plethora of fast interprocess synchronization
> and communication mechanisms, high resolution clocks/timers, extended
memory
> mapping capabilities, and the ability to shield processors from interrupts
> and dedicate processes to specific processors in order to guarantee
> deterministic system response, etc. It is currently based on the 2.4.21
> kernel from kernel.org. It is installed by first installing Red Hat 8.0 on
> the target system, and then installing the RedHawk kernel and real-time
> libraries, etc. It is Red Hat 8.0 compatible. A propaganda page is here
> http://www.ccur.com/isd_solutions_redhawklinux.asp if anyone wants more
> info.)
> 
> The work I've done so far includes:
> 
> 1) Create a "redhawk-x86" build environment.
> 
> 2) Moved a few elements of the "posix" source files to the new "redhawk"
> branch of the source tree and modify them to better take advantage of some
> RedHawk features. The work done does not affect a lot of code, and it
> remains largely based on the posix base. Most of the work is as follows,
> though there some additional bits and pieces:
>     - osdMutex: now uses RedHawk fast mutex mechanisms.
>     - osdThread: has now enabled the existing
> _POSIX_THREAD_PRIORITY_SCHEDULING code. This was not sufficient to enable
> priority scheduling under RedHawk, so a small amount of extra code was
> required (e.g. specifically enabling "SCHED_RR" [or SCHED-FIFO] on a
> per-thread basis).
>     - osdTime: Now uses CLOCK_REALTIME_HR for highest possible resolution.

> 

 From what I answered above, I agree with the idea of creating a new 
subdirectory  libCom/osi/os/redhawk

Eric and I also mentioned the priviledge problem.

> 3) Compile the ioc base with all the changes and run the "epicsXXXXTest"
> programs.
> 
> I would appreciate advice in two areas: some specific source code
questions,
> and then an initial much higher level question. (My biggest problem at the
> moment is that I'm "looking at the wood, not the trees" - that is, I'm
> hacking bits of low-level code without having much of an understanding of
> how epics works as a whole. I have, however, read through the EPICS I/O
> Controller Application Developer's Guide, some other EPICS training-type
> docs from the web, and some whitepapers/technical reports which give an
> overview of porting EPICS. Overall, these have been very helpful.....)
> 
> Firstly, I'd appreciate some help with code-specific queries:
> 
> a) The posix/osdThread.c routines uses "pthread_mutex_XXXX()" functions
for
> synchronization, rather than the platform specific mutexes in osdMutex.c.
Is
> there any reason for this ? (I've changed osdThread.c to use the mutexes
in
> osdMutex.c and everything seems OK so far...)
> 

I answered this above. I agree with your approach.

> b) The posix/osdTime.cpp file appears to contain current references to
> "gmtime_r()" and "localtime_r()" functions, which I gather are specific to
> VxWorks. These do not appear to be resolved anywhere in a build under
Linux
> (or RedHawk). Is this correct, or am I missing something ?
> 

These are defined in the Single Unix Specification Version 3 (The successor
to 
POSIX standards). See
http://www.unix.org/single_unix_specification/

> c) I don't yet understand the way the thread priorities are calculated in
> posix/osdThread.c, when _POSIX_THREAD_PRIORITY_SCHEDULING is enabled. Can
> someone please explain to me the effect that the getOssPriorityValue()
> function is intended to have ?
> 
getOssPriorityValue()

Given a epicsThread priority return a OS specific priority for it.

epicsThread priorities are from 0 to 99 with a higher number meaning higher 
priority.

On vxWorks priorities are from 0 to 255 with a lower number meaning higher 
priority. For the vxWorks implemenation we decided to map epics priorities 
0,...,99 to vxWorks priorities 199,...100.

> d) Under RedHawk, if a process desires to run with a real-time priority
> (i.e. use a real-time scheduling policy of SCHED_RR or SCHED_FIFO, where
the
> real-time priorities range from 99 down to 1 and are all therefore higher
> than the standard Linux SCHED_OTHER timesharing priority of zero), then
the
> process must be owned by, or run in the context of, the root user. I could
> do this by logging on as root and MAKEing the code as root. However, I
> prefer to MAKE the project in the context of my normal user login, which
> means that I then need to "chown root XXXX" and "chmod a+s XXXX" all the
> created binary executables so that they run as the root user. I'd like to
be
> able to automate the "chown" and "chmod" as part of the build (MAKE)
> process, but I haven't yet figured out how to modify the build environment
> to do this. Can anyone suggest how to do this, or at least where I should
be
> looking ?
> 

I suggest that you use SCHED_FIFO. This is what we do for vxWorks.
The problem of root was discussed above and by Eric Norum's message.

> e) What other areas of the code should I be looking at, in order to
improve
> the real-time performance (in a Linux-type environment) ?
> 

Everywhere :-)

> Now for the "trees"-type question....
> 
> i) I'm doing this work on my laptop computer, with some further testing on
a
> multi-processor system in our office. In this restricted environment,
what's
> the simplest way to test that the resultant EPICS "application" runs
> correctly ? In the same vein, what's the simplest way to run a
> performance/response-time test of the EPICS/ioc code ? (I'd like to run
some
> performance/response measurements under standard Linux and then again
under
> RedHawk. Am thinking of doing something like that described in the
"Porting
> EPICS to L4-Linux Based System" whitepaper, where a thread grabs the CPU
> counter, kicks an I/O module into interrupting, again grabs the CPU
counter,
> and then measures the difference in the counter values....)
> 

Perhaps Odagiri-San ([email protected]) could comment. He is the one
who 
did the whitepaper.

> Thanks for any and all advice.
> 
> Kind regards,
> 
> David Kelly

Marty Kraimer

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System