EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  <20042005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  <20042005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RE: Buffer problems
From: "Kevin M. Peterson" <[email protected]>
To: [email protected]
Date: Tue, 06 Apr 2004 11:45:54 -0500
Three of UNICAT's IOCs hung at the same time yesterday. The three crates had been
idle (nothing had been written to the log files for at least 5 hours). The three log files
show that the three IOCs (LGA, PIE, and FAT) experience the same error:


LGA: 2004-04-05,16:07:12: 0x760408 (tNetTask): ei driver: command field frozen
PIE: 2004-04-05,16:07:12: 0x760408 (tNetTask): ei driver: command field frozen
FAT: 2004-04-05,16:07:28: 0xf60408 (tNetTask): ei driver: command field frozen


Each of the log files shows that the ENOBUFS and save/restore errors occurred 1-3 minutes
later and continued until each IOC was rebooted. Here are the errors that occurred:


CAC: unexpected select fail: 851971=S_iosLib_INVALID_FILE_DESCRIPTOR
0x2174b4 (save_restore): save_file: Backup file (auto_sr/settings.savB) is bad. Writing a new one.
0x32d250 (CA_online): ../online_notify.c: CA beacon error was "S_errno_ENOBUFS"
CAC: error = "S_errno_ENOBUFS" sending UDP msg to 164.54.216.255:5064
0x2174b4 (save_restore): save_file - unable to open file auto_sr/settings.savB
0x2174b4 (save_restore): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
0x2174b4 (save_restore): save_restore:save_file: Can't write new backup file. I quit.
0x2174b4 (save_restore): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
sendto failed: S_errno_ENOBUFS
CAC: error = "S_errno_ENOBUFS" sending UDP msg to 164.54.216.255:5064
sendto failed: S_errno_ENOBUFS


We have seen the ENOBUFS errors occur in IOCs before without seeing the tNetTask error,
however, in those cases only one IOC was affected.


This is the first time we have seen both the tNetTask error and multiple IOCs have ENOBUFS
problems simultaneously.


Are these "command field frozen" errors occuring anywhere else?


Kevin Peterson



At 12:42 PM 3/29/2004 -0600, Mark Rivers wrote:
Hi Ernest,

Thanks for the reply.

I really don't think it's anything with CA.  These IOCs were running
fine for a long time, years in some cases.  Suddenly last week they all
hung up at the same time.  I am able to completely reproduce the problem
by simply sending 2000 1500-byte ping packets to them as fast as
possible.  So I am quite sure that some computer on the network is doing
the equivalent of that.  It is happening every few minutes or hours. The
problem is that the NSLS is one giant subnet with 512 addresses.  If we
disconnnect the switches at the beamline from the NSLS subnet then
things are fine, so it's not a problem with the local CA server and
clients.  They need to get a sniffer on the network and find who the
culprit is.  More importantly, Wind River needs to fix their network
software so that sending a 2000 1500-byte packets does not crash the
network.

Mark



Navigate by Date:
Prev: Re: undefined symbols during download app.munch Andrew Johnson
Next: RE: Buffer problems Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  <20042005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: RE: Buffer problems Mark Rivers
Next: RE: Buffer problems Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  <20042005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 10 Aug 2010 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·