Subject: |
Large number of sequencers running per soft IOC under Linux: Kernel becomes unresponsive |
From: |
"Ernest L. Williams Jr." <[email protected]> |
To: |
Tech-talk <[email protected]> |
Date: |
Wed, 27 Apr 2011 08:18:27 -0700 |
Hi,
When reviewing the recent "gateway aborted by SIGBUS" EPICS tech-talk
thread; I recall a possibly related issue.
Here I am referring to understanding what system resources are impacted
as the number of softIOC(s) grows
per "linux server"
We had an incident in which there were 11 linux-based softIOCs each
running 1190 EPICS sequencers.
When only 10 of the softIOCs were running the linux system was responsive.
However, when the 11th IOC was launched we came to a "screeching halt". :(
We could not remotely log in; even at the console we could do nothing.
The only way to recover was to reboot the linux server.
Our computer systems group noticed that the default number of "file
descriptors" per process system wide was 1024.
Our team also monitored many of the linux system resources during the
instantiation of the 11 softIOCs.
When 10 softIOCs are running each with 1192 EPICS sequencers the linux
system wide resources seemed to be no where near
the limit. That is we monitored CPU load , Memory, I/O Activity, etc...
When the number of file descriptors per process was increased to 2048
the problem remained.
An increase in the number of file descriptors per process to 4096 also
failed.
In the end, the computer systems group increased the default number of
file descriptors per process to "8192" and the problem was solved.
We still do not understand the relationship between the linux system
resources and how are softIOCs w/sequencers drive us into some
linux system limitation.
When looking at the number of file descriptors for a single softIOC, it
is well below the default limit.
Using iocAdmin EPICS Module we look at: <>:FD_MAX and <>:FD_FREE
=============================================================
SIOC:LI25:KY00:FD_MAX = 8192
SIOC:LI25:KY00:FD_FREE = 8173
Appears that only 19 FD(s) are being used. So, I am at a lost.
=============================================================
Has anyone else seen an issue like this?
Any ideas and references will be appreciated.
- Navigate by Date:
- Prev:
Handle leak in Channel Access Server V4.13 (current EPICS base) - WinXP Carsten Winkler
- Next:
areaDetector Prosilica IOCs on Windows 7? Mark Rivers
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
<2011>
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
- Navigate by Thread:
- Prev:
RE: Handle leak in Channel Access Server V4.13 (current EPICS base) - WinXP Jeff Hill
- Next:
areaDetector Prosilica IOCs on Windows 7? Mark Rivers
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
<2011>
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
|