EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <20112012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <20112012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Large number of sequencers running per soft IOC under Linux: Kernel becomes unresponsive
From: "Ernest L. Williams Jr." <[email protected]>
To: Tech-talk <[email protected]>
Date: Wed, 27 Apr 2011 08:18:27 -0700
Hi,

When reviewing the recent "gateway aborted by SIGBUS" EPICS tech-talk thread; I recall a possibly related issue. Here I am referring to understanding what system resources are impacted as the number of softIOC(s) grows
per "linux server"

We had an incident in which there were 11 linux-based softIOCs each running 1190 EPICS sequencers. When only 10 of the softIOCs were running the linux system was responsive. However, when the 11th IOC was launched we came to a "screeching halt". :(
We could not remotely log in; even at the console we could do nothing.
The only way to recover was to reboot the linux server.

Our computer systems group noticed that the default number of "file descriptors" per process system wide was 1024.

Our team also monitored many of the linux system resources during the instantiation of the 11 softIOCs. When 10 softIOCs are running each with 1192 EPICS sequencers the linux system wide resources seemed to be no where near
the limit.  That is we monitored CPU load , Memory, I/O Activity, etc...

When the number of file descriptors per process was increased to 2048 the problem remained. An increase in the number of file descriptors per process to 4096 also failed.

In the end, the computer systems group increased the default number of file descriptors per process to "8192" and the problem was solved. We still do not understand the relationship between the linux system resources and how are softIOCs w/sequencers drive us into some linux system limitation. When looking at the number of file descriptors for a single softIOC, it is well below the default limit.
Using iocAdmin EPICS Module we look at: <>:FD_MAX and <>:FD_FREE
=============================================================
SIOC:LI25:KY00:FD_MAX = 8192
SIOC:LI25:KY00:FD_FREE = 8173
Appears that only 19 FD(s) are being used.  So, I am at a lost.
=============================================================
Has anyone else seen an issue like this?
Any ideas and references will be appreciated.

Navigate by Date:
Prev: Handle leak in Channel Access Server V4.13 (current EPICS base) - WinXP Carsten Winkler
Next: areaDetector Prosilica IOCs on Windows 7? Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <20112012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: RE: Handle leak in Channel Access Server V4.13 (current EPICS base) - WinXP Jeff Hill
Next: areaDetector Prosilica IOCs on Windows 7? Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  <20112012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 18 Nov 2013 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·