Allan,
It does sound like optical telescope upgrades are a tad bit more
challenging, and schedule risk exposed, compared to what we typically see in
the particle accelerator world.
We have actually made the network behavior of EPICS systems as a
whole more robust under load in R3.14. The main change has been that network
congestion induced disconnects do _not_ cause positive congestion
feedback. In practice this means that the client library disconnects the
application, but not the TCP circuit when a CA channel times out. A large
number of other subtle issues have also been fixed, and KECK can quickly
benefit from many of them by switching all the apps to R3.14 even if the IOCs
must stay at R3.13 because it takes more time to complete their upgrade work.
Ø We have
deployed r3.14 caRepeaters
I wouldn’t expect any functionally different behavior between
the R3.14 and R3.13 CA repeaters, but I am very willing to have a look at some
additional details to see what might be occurring. At some point we did change
from using the fork system call to auto-start the CA repeater to using the exec
system call which is certainly a behavior change, but I seem to recall that
this change was made as a patch at some point to R3.13. The big change here is
avoidance of duplicating resources created by an application into the CA
repeater’s process, and problems with the CA repeater process having a strange
name belonging to the app that stated it. One can of course avoid such issues
altogether by starting the CA repeater in the workstation’s boot script.
Ø And we
have some random issues: sometimes we have ‘get’ failure
Ø responses,
indicating channels are disconnected or non-existant,
Ø in one
client, whilst other clients are happy;
This can certainly occur if there is CPU saturation in the IOC,
buffer starvation in the IOC’s IP kernel, or possibly also if you have a really
old network with hubs instead of switches. Old Ethernet networks can experience
delays if you have collision chains of sufficient magnitude. Another
possibility is applications with too-short connection timeouts configured. Here
at LANL I have, only on one project, seen a weird situation where many HP CAD
workstations were configured with the wrong network/host partitioning mask, and
they were responding to CA search requests with an ICMP error response. That
was causing the IP input queue on the CA client’s host to get saturated, and
this led to CA connects taking much longer to complete than they should have.
Otherwise not expected.
Ø responses
from multiple IOCs for a given channel when only one of
Ø those IOCs
has that channel.
Not expected, and I don’t recall hearing complaints in this area.
If the CA client library received a search response from the wrong IOC it will
try to connect to the wrong IOC, fail doing that, and then return to searching
for the channel where it would presumably eventually find the correct IOC.
Ø I have not
had time to delve into those. The errors seem
Ø to be
realtime IOC/board and/or possibly vxWorks related, as there
Ø is one
newer processor (not PPC as are most of the others) running
Ø a newer
version of vxWorks, from which I never see those types of errors.
Some earlier vxWorks had substantial issues with mbuf starvation
and driver buffer pool starvation. As I recall, the SNS resolved these issues
by allocating more m-bufs and cluster-bufs at vxWorks kernel build time, and by
installing all of the very latest vxWorks network interface driver patches. The
network congestion robustness improvements in R3.14 probably helped also.
Ø Does
someone have a simple multi-threaded example, utilizing r3.14
Ø (so I can
compare the CA library calls with what we are currently using in r3.13.10)?
The CA client interfaces are very close to 100% backwards
compatible. There are some new interfaces that enable new features of course.
Hopefully I am not oversimplifying the situation, but it’s probably safe to say
that the primary multi-threading issue will be with how to properly structure
your app for multi-threading as all of the multi-threading issues related to CA
internals have been dealt with when preparing the first releases of R3.14. As
mentioned in previous mm, you will also need to decide if you want
non-preemptive callback which requires periodic calls to ca_poll from your
thread, or non-preemptive callback where you will receive asynchronous
callbacks from the CA client library. Asynchronous callbacks will of course
require some additional expertise. The application may need some additional
mutual exclusion primitives to control asynchronous access into the
application’s data structures originating from multiple instances of the CA
client library’s auxiliary threads.
Jeff
______________________________________________________
Jeffrey O. Hill
Email [email protected]
LANL MS
H820
Voice 505 665 1831
Los Alamos NM 87545 USA
FAX 505 665 5107
Message
content: TSPA
No negativity noticed J
We have deployed r3.14 caRepeaters. And we have some random issues:
sometimes we have ‘get’ failure responses, indicating channels are disconnected
or non-existant, in one client, whilst other clients are happy; responses from
multiple IOCs for a given channel when only one of those IOCs has that channel.
I have not had time to delve into those. The errors seem to be realtime
IOC/board and/or possibly vxWorks related, as there is one newer processor (not
PPC as are most of the others) running a newer version of vxWorks, from which I
never see those types of errors.
I think our big issue, with respect to forging forward with
multi-thread clients, (i.e. using r3.14 for all clients) is that major
modifications would need to be made to the layer we have between CA and our
clients (said layer hides CA, as it is not the only inter-process/processor
communications mechanism in place, for instance we have numerous RPC systems;
and other socket based systems). Most of our operational clients do not
interface directly to CA. Hence, that ‘layer’ is critical. It was the
application interface provided to all our sister institutions (which create
non-EPICS/CA instruments/systems) back in the early 90’s. I have been studying
that ‘layer’ in great detail, in my attempt to solve the multi-threaded issue
(r3.13.10), and it may be that I am now sufficiently less ignorant that I can
make those modifications. If that is the case then I will no doubt have more
questions. So, thanks for pointing me to pertinent documents that will
make the transition form r3.13.10 to r.3.14 possible.
Does someone have a simple multi-threaded example, utilizing r3.14
(so I can compare the CA library calls with what we are currently using in
r3.13.10)?
Cheers,
Al
Aloha again Allan,
Sorry, after rereading my message, the tone sounds a bit
negative which wasn’t my intent. I should have said, “please read also the
section in the reference manual entitled - Thread
Safety and Preemptive Callback to User Code”. When designing this type
of application, one must decide if CA callbacks should occur only when
periodically executing in a CA client library function such as ca_poll, or if
the CA callbacks should occur asynchronously, as soon as the network messages
are processed by the auxiliary threads in the library. Either approach can be
used in a multi-threaded program.
Jeff
______________________________________________________
Jeffrey O. Hill
Email [email protected]
LANL
MS H820
Voice 505 665 1831
Los Alamos NM 87545 USA
FAX 505 665 5107
Message content: TSPA
Aloha
Allan,
Ø Does the seg fault
occur because r3.13.10 is NOT thread safe?
The R3.13 CA Client library is definitely __not__ thread safe, and
I can easily imagine that this might be the cause of your seg fault.
Ø Does anyone have an
example of a multi-threaded app using r3.13.10 on UNIX?
The R3.14 CA client
library _is_ thread safe, and it should also interoperate fine with
R3.13 IOCs. We routinely operate LANSCE with that configuration in our
production system. Our control room runs R3.14, but many of our IOCs still run
R3.13. You should read the section in the reference manual entitled “Thread
Safety and Preemptive Callback to User Code“.
Jeff
______________________________________________________
Jeffrey O. Hill
Email [email protected]
LANL
MS
H820
Voice 505 665 1831
Los Alamos NM 87545 USA
FAX 505 665 5107
Aloha
I
am trying to get a multi-threaded application working on SunOs 5.10 with
connection to two UNIX IOC’s.
I
get a seg fault for ellDelete, two statements from the end of cac_select_io()
(epics/r3.13.10/base/src/ca/bsd_depen.c).
The
seg fault does not occur immediately but within a couple of minutes
(connections are to two IOC’s running on UNIX, with events from two long
records on each IOC, where one record on each system is updated at 1 hz and the
other at 10 hz).
Does
the seg fault occur because r3.13.10 is NOT thread safe?
Does
anyone have an example of a multi-threaded app using r3.13.10 on UNIX?
Thanks,
Allan