EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  <19971998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  <19971998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: hanging IOCs
From: Marty Kraimer <[email protected]>
To: [email protected]
Date: Wed, 19 Feb 1997 08:18:20 -0600
Ric Claus wrote:
> 
> We (SLAC's PEP-II RF group) are having a problem with hanging/crashing
> IOCs and wonder if anyone has some suggestions.  We're running EPICS
> R3.13.0.beta4 with dm version 2.3.  The processor is a National
> Instruments VXIcpu-030 with 8 MB of RAM.  When everything is up and
> running normally we see about 2 MB of free memory.  Task stacks are not
> close to the edge.  There is no VXI interrupt activity, although perhaps
> some are being genterated by the Allen Bradley scanner module.

I do wonder about the memory. If lots of CA clients connect the 2MB
could go quickly. However CA stops accepting new clients when free
memory gets too low. You should see error messages if this happens.

How often do IOCs hang/crash?
Have you ever been able to issue vxWorks commands to an IOC that appears
to be hung or crashed?
If so run spy, memShow, and i. If any tasks are suspended run tt.
If you have a particular IOC that hangs frequently could you
periodically
issue memShow, i,and spy and look for changes.

> Occasionally we see the Allen Bradley driver spontaneously changing the
> status of all connected links from good to bad.  Sometimes they go back
> to good a moment later, sometimes it's a long while later, sometimes not
> at all.  There is nothing an operator can do to cause this directly.
> The blue hose is not being unplugged or anything like that.  We're
> using a 6008-SV1 scanner module.

It is the scanner itself that detects that a link is down. The driver
just asks the scanner periodically for a link status report and uses the
result.

A couple of things to try.

Check all "blue hose" connectors and terminators for signal integrety.

Does the ab_reset command bring back a "dead" system? This command is a
"last resort" command but may bring the system back.

Is there any possibility of EMF noise disturbing the blue hose?

> When an IOC crashes or hangs, there is generally nothing to see on the
> console port record.  Once, however, I saw tNetTask trying to say
> something, but it never made it out.

Again if console port accepts vxWorks commands try commands listed
above.
 
> One possibility I wonder about is that CA clients cause the creation
> of "CA client" and "CA event" tasks that consume a lot (800 Mb) of
> memory.  Perhaps these crashes are due to one too many dm sessions
> being fired up in people's offices.  Is there a way to limit the
> number of CA clients?  If not, shouldn't such a thing be added?
> What do these tasks do with that much memory?  When IOC resources
> run out, does CA currently stop allowing new client connections or
> does it let the IOC die?  How do other people handle this situation?
> Just add more memory?

Again CA refuses new connections when memory gets low. Even if memory
gets low already connected clients useyally stay connected. If this was
cause of your problem I think you would have seen more than just a hard
crash.

Last minute info. Ned Arnold just stopped by. He is going to send a
message about VME modules in VXI systems. He may have a reason for the
AB link failures.

Marty Kraimer


References:
hanging IOCs Ric Claus

Navigate by Date:
Prev: Re: R3.13 Chip Watson
Next: Re: hanging IOCs Ned Arnold
Index: 1994  1995  1996  <19971998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: hanging IOCs Chip Watson
Next: Re: hanging IOCs Jeff Hill
Index: 1994  1995  1996  <19971998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 10 Aug 2010 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·