Experimental Physics and
Industrial Control System

Marty Kraimer <[email protected]> · Wed, 10 Dec 2003 07:39:09 -0600

When you are done with your redhawk port it can be put into base if a
responsible person from your company provides a signed release form stating that
your contribution can be subject to the EPICS Base License.
David Kelly wrote:

Thanks Marty.
I've read all the follow-up correspondence, especially those from yourself,
Jeff Hill, Andrew Johnson, and Eric Norum. If I may reply to some of the
points raised:
1) I still intend to create a Redhawk-specific version of osdMutex and
osdThread, etc. This seems to be the most easily understandable and most
maintainable solution...though I'm open to better ideas. I intend that the
Redhawk-specific version of osdMutex will only contain one implementation of
a mutex.

A agree with this solution. I also want the same thing for the Linux 2.6 kernel.
It is on my TODO list but I have been waiting until an official 2.6 release is
available from one of the major Linux distributions.
Ulimately ALL supported platforms should support thread priorities. Perhaps when
all support platforms have had time to develop a good and complete
implementation of pthreads then we can go back to a common pthreads
inplementation. For now I think it is better provide a separate implementation
of osdThread for each platform in order to implement thread priorities. A
separate implementation of osdMutex may also be desirable.
2) Having read the discussion about mutexes and spinlocks, etc, I'm
convinced that the osdMutex code that I've built already, needs to be
revised. At present, the code is based on "true" spinlocks (i.e. initially,
an atomic memory operation instruction, followed by spinning using a
"standard" memory read operation and then retrying the atomic lock when the
"read" indicates that the lock is available), although it is cognisant of
whether it is running on a system which has 1 CPU active, or >1 CPU active
(including logical CPUs enabled by Hyperthreading). The present code only
spins if it's running on a multi-CPU system, otherwise it calls
"sched_yield()" to give up the CPU as after it tries once, and fails, to
acquire the lock.

But this only yields to threads that have the same or higher priority.
At present I do nothing special to cope with the Priority
Inheritance problem - I confess that I don't know enough about the EPICS
code to understand if Priority Inheritance is really required, or if the
code can never exhibit such problems. However, having said that, I suspect
that the code should somehow be made to cater for PI. (A forthcoming release
of Redhawk will provide kernel support for PI on mutexes, so in that case I
could spin for a while and then use a kernel mutex - but I don't want to
have to wait for it...) I'll let you know what I wind up building.

For IOCs recursive mutexes are mandatory. Priority Inheritance is nice but not
mandatory.
3) Re the difference between Redhawk and the new Linux 2.6 (presumably the
"3.6" in your email was a typo) kernel...  My understanding (not gospel) is
that the following Redhawk features will come as standard to all Linux
systems in the 2.6 kernel:
  - kernel preemption
  - low-latency kernel patches (though these are only a subset of what's in
Redhawk)
  - the real-time scheduler
  - Posix timers (though only "low" resolution, not close to Redhawk's high
res timers)
This means that (at least) the following existing Redhawk features will not
be present in the standard Linux 2.6 kernel:
- rescheduling variables

What is a rescheduling variable?
- fast block/wake services

I will guess that these will be tricky to make more efficient than what the
regular 2.6 kernel provides.
- usermap and /proc enhancements

Couldn't these be done by user level code on top of standard Linux facilities?
- processor shielding and affinity
- frequency-based scheduler
Then there are new features planned for future releases of Redhawk and, as
mentioned, one of these is support for Priority Inheritance.

5) I'm not sure what to do about the "must run as root" problem. I can't use
a wrapper as Eric suggested, because (for example) I can't set the
scheduling policy and priority for all threads once - at program
commencement - because this must be done as each thread is created or after
a thread is created. I'm not sure how to handle this problem - I *might*
have to really run stuff as root. I know Redhawk supports Pluggable
Authentication Modules (PAMs), but I've never looked at them in detail, so
I'm not sure if they provide the functionality required. Will have to take a
look... As always, other suggestions are welcome...

This is really important. It is a problem that must be solved in a nice way. I
know that at APS if we don't have a good solution our Network Administrators
will not be happy.
Marty Kraimer

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System