EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  <20072008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  <20072008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: channel access security deadlock from asInit()
From: "Ned D. Arnold" <[email protected]>
To: Ralph Lange <[email protected]>
Cc: Kay-Uwe Kasemir <[email protected]>, tech talk <[email protected]>
Date: Wed, 18 Apr 2007 11:58:18 -0500
Ralph -

I think the behavior that you observed is by design ... if one tries to specify a configuration file that does not exist, it means you intended to have *some* access control in place. If the file doesn't exist, this is a fatal error and the safest thing to do is not allow any access at all so someone will fix the problem immediately.

I actually remember this discussion with Marty Kraimer about 16 years ago ...

Ned


Ralph Lange wrote:
We have seen a very similar behaviour just a day ago, when the st.cmd script is pointing to an AS configuration file that doesn't exist. The CA server seems to hang, clients do not get any PV names on the IOC resolved.
Rebooting with a valid file in place solves the problem.
I don't think a network event is part of the game here - but I wouldn't bet. No Java CA involved - that I'm sure about.


Thanks a lot for doing the analysis - we just don't have the time right now....

Ralph


Kay-Uwe Kasemir wrote:
Hello:

We're having problems with IOCs deadlocking when re-loading the
Channel Access Security configuration files at runtime.
Maybe somebody else has seen this, too?

When doing this via the subroutine record AS routines,
a follow-up 'asDump' on the IOC shell will show that the file
was indeed loaded OK, but no more connections are allowed.
When instead running 'asInit' from the IOC shell, that hangs,
and a CTRL-C shows a stack trace that's analyzed below.

This happens with both R3.14.7 and R3.14.8.2,
on IOCs that do very different things.
After an IOC reboot, 'asInit' calls work fine,
and the access security files are of course OK as well.

By looking at the affected IOCs and their uptime, we assume
that this is caused by something that happened on the SNS
network between March 13 and April 4.
In the past, we suspected the 'pure Java CA client' code
to cause CA security related deadlocks. In principle,
we don't expect any more pure Java CA client code on the
SNS controls network, but we can't be 100% sure.
On the other hand, this deadlock could be completely unrelated
to Java CA, so forget all I said about Java CA.

Anyway, here's the stack trace with guesses based on
about 2 hours of comparing the stack offsets with the vxWorks 'l'
output and the 'gcc -S' PowerPC assembly code:

1e1a54 yystart        +96c: asInit ()
1b5b1a8 asInit         +10 : 1b5ab3c () = asInitCommon()
1b5ac28 asTrapWriteRegisterListener+384: asInitFile ()
1b59564 asInitFile     +7c : asInitFP (34a7)
1b545dc asInitFP       +11c: asInitialize ()
1b542b0 asInitialize   +31c: 1b55ec0 ()
  This is asAddMemberPvt()

1b56018 asDumpMemFP    +49c: 1b561cc ()
  This is asComputePvt()

1b563c8 asDumpMemFP    +84c: 1b650dc ()
  This must be the change-of-access-rights callback
  at the end of asComputePvt():
    if(pasgclient->pcallback && oldaccess!=access) {
        (*pasgclient->pcallback)(pasgclient,asClientCOAR);

1b6519c casUserNameInitiatingCurrentThread+1ebc: 1b64f5c ()
Based on a grep of the sources, the only code that registers for such
a callback is in rsrv/camessage.c, so this must be casAccessRightsCB(),
which calls epicsMutexMustLock(pclient->eventqLock)


1b6502c casUserNameInitiatingCurrentThread+1d4c: epicsMutexLock ()
1a7a7c0 epicsMutexLock +24 : semTake ()
1f0c74 semTake        +13c: semMTake ()
  Deadlock...


-Kay



Replies:
Re: channel access security deadlock from asInit() Ralph Lange
References:
channel access security deadlock from asInit() Kay-Uwe Kasemir
Re: channel access security deadlock from asInit() Ralph Lange

Navigate by Date:
Prev: Re: channel access security deadlock from asInit() Ralph Lange
Next: Re: channel access security deadlock from asInit() Ralph Lange
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  <20072008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: channel access security deadlock from asInit() Ralph Lange
Next: Re: channel access security deadlock from asInit() Ralph Lange
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  <20072008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 10 Nov 2011 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·