Experimental Physics and
Industrial Control System

Jeff Hill <[email protected]> · Mon, 23 Aug 2010 09:25:36 -0600

Hi Dirk,

Sorry about the slow response, I was away on vacation Friday.

> There is nothing to trace. The process is not there anymore. The
> gateway process just ends without any message. Just as if something has
> called exit().

You might try attaching gdb to the gateway process before it crashes, as 
follows. This can produce results quickly, and in the past my suspicion has 
been that better results are obtained when running gdb against a live
process
when compared with working against a core file.

"gdb <full path to image> <process id>"

And, an additional benefit is that if in fact one of the threads is calling
exit, 
that could be readily detected by typing "b exit" which will set a
breakpoint on 
the exit function.

Thanks for your help,

Jeff
______________________________________________________
Jeffrey O. Hill           Email        [email protected]
LANL MS H820              Voice        505 665 1831
Los Alamos NM 87545 USA   FAX          505 665 5107

Message content: TSPA

> -----Original Message-----
> From: Dirk Zimoch [mailto:[email protected]]
> Sent: Friday, August 20, 2010 4:10 AM
> To: Jeff Hill
> Cc: [email protected]
> Subject: Re: CA gateway dies without error message
> 
> Hi Jeff,
> 
> There is nothing to trace. The process is not there any more. The
> gateway process just ends without any message. Just as if something has
> called exit().
> 
> My start line is:
> export EPICS_CA_MAX_ARRAY_BYTES=4000000
> ulimit -s 2048
> export EPICS_CAS_BEACON_ADDR_LIST=172.21.10.255
> caGateway -no_cache -archive -prefix FIN-CCAGW11W-EXP \
> 	-sip 172.21.10.22 \
> 	-cip 172.21.11.255 \
> 	-uid 922 -gid 922 \
> 	-pvlist GATEWAY.pvlist \
> 	-access GATEWAY.access \
> 	-log gateway.log \
> 	"$@" > gateway.err 2>&1 < /dev/null &
> 
> Gateway startup log:
> Aug 19 15:32:36 PV Gateway Version 2.0.4.0 [Oct 16 2009 10:54:59]
> EPICS 3.14.11 PID=4757
> EPICS_CA_ADDR_LIST=172.21.11.255
> EPICS_CA_AUTO_ADDR_LIST=NO
> EPICS_CA_SERVER_PORT=Not specified
> EPICS_CA_MAX_ARRAY_BYTES=4000000
> EPICS_CAS_INTF_ADDR_LIST=172.21.10.22
> EPICS_CAS_SERVER_PORT=Not specified
> EPICS_CAS_IGNORE_ADDR_LIST=Not specified
> Running as user cagw on host fin-ccagw11w
> Statistics PV prefix is FIN-CCAGW11W-EXP
> 
> There is nothing in gateway.err and no error message in gateway.log
> with
> a time stamp near the crash. The usual goodbye message like "Jul 13
> 17:12:40 PV Gateway Ending (SIGTERM)" is missing.
> 
> I will try to enable core dumps. Haven't done that yet.
> 
> Dirk
> 
> Jeff Hill wrote:
> > Hi Dirk,
> >
> > I did put some considerable effort into improving the quality (in
> terms of
> > efficiency, stability, performance, and functionality) of the PCAS
> and the
> > gateway around this time last year. And I believe that the result was
> a
> > significant step forward. Which version of the gateway are you
> running?
> >
> > Currently, I have no clues on your issue to start with so it's hard
> to move
> > forward. So I will 2nd Andrew's suggestion of attaching gdb and
> getting a
> > stack
> > trace of the failed thread. One can attach gdb to a running process
> by
> > typing
> > "gdb <full path to image> <process id>". If gdb can't be attached to
> the
> > running process then another alternative is to attach gdb to the core
> > file using "gdb  <full path to image> <full path to core file>".
> > In gdb type first "bt" to show the
> > thread that crashed and then follow that with "thread apply all bt"
> so that
> > I know what all threads are up to when the crash occurred. Lots of
> output
> > but
> > exactly what is needed to get the matter resolved. This works better
> if the
> > code is compiled for debugging, but I might be able to resolve the
> issue
> > with
> > only the stack traces from the optimized version (which runs faster
> and so
> > can have a different set of failure scenarios) - so I am very willing
> to
> > look at traces from an optimized version first if it will save some
> effort
> > on your end.
> >
> > Thanks in advance for your help,
> >
> > Jeff
> > ______________________________________________________
> > Jeffrey O. Hill           Email        [email protected]
> > LANL MS H820              Voice        505 665 1831
> > Los Alamos NM 87545 USA   FAX          505 665 5107
> >
> > Message content: TSPA
> >
> >
> >> -----Original Message-----
> >> From: [email protected] [mailto:tech-talk-
> >> [email protected]] On Behalf Of Dirk Zimoch
> >> Sent: Thursday, August 19, 2010 7:52 AM
> >> To: EPICS
> >> Subject: CA gateway dies without error message
> >>
> >> Hi all,
> >>
> >> Since upgrading the EPICS base version of the caGateways to 3.14.11,
> >> the
> >> gateway dies from time to time for no obvious reason. It does not
> write
> >> any log message when it dies (as it would do in a clean shutdown, in
> >> case of a signal or when any strange CA event happens.) It also does
> >> not
> >> write anything to stderr. I cannot remember that I have seen this
> type
> >> of fault with 3.13.8.
> >>
> >> Any idea?
> >>
> >> Dirk
> >
> >

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System