EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  <20092010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  <20092010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Gateway fails with "epicsMutex pthread_mutex_lock failed: error Invalid argument"
From: Mark Bennett <[email protected]>
To: "[email protected]" <[email protected]>
Date: Thu, 8 Jan 2009 12:24:09 +1100
One of our EPICS gateways appears to be failing when an IOC is restarted multiple times within half an hour or so. A large number of PVs (4,000) are accessed via this EPICS gateway - by a channel archiver specifically. Prior to failure, the client event and post rates are equal at ~1550Hz while the server event rate is about half the post rate (server post rate ~1550Hz).

A large number of gateways (12) are hosted on a single box in separate directories. The box is running 64-bit redhat linux but we are currently building linux-x86 binaries. The number of available threads has been increased by decreasing the stack size using "ulimit -s 2048". The gateway is started using options -no_cache and -archive. We followed Dirk Zimoch's guidelines from a previous post (thanks).

We are using EPICS base 3.14.9 and gateway 2.0.3.0.

The IOC is on network 10.135 and the archiver is on network 10.100. The IOCs IP (restarted multiple times from above) is 10.135.2.11. There are other IOCs on this subnet whose PVs are accessed via the same gateway:

    IOCs on 10.135 ---> EPICS gateway (10.135 --> 10.100) ---> Channel Archiver 10.100.0.244

Here is an example gateway error log - the gateway process does not die but becomes non-functional:

Jan 07 16:00:21 PV Gateway Version 2.0.3.0 [Dec 17 2008 18:16:17]
EPICS 3.14.9 PID=16488 ServerPID=16487
EPICS_CA_ADDR_LIST=10.135.255.255
EPICS_CA_AUTO_ADDR_LIST=NO
EPICS_CA_SERVER_PORT=Not specified
EPICS_CA_MAX_ARRAY_BYTES=5000000
EPICS_CAS_INTF_ADDR_LIST=10.100.10.247
EPICS_CAS_SERVER_PORT=Not specified
EPICS_CAS_IGNORE_ADDR_LIST=Not specified
Running as user ics on host CR01GW04
Statistics PV prefix is SR12ID01GW04
Jan 07 16:02:21 Warning: Virtual circuit disconnect 10.135.2.11:5064
Jan 07 16:08:28 Warning: Virtual circuit disconnect 10.135.2.11:5064
Jan 07 16:29:23 Warning: Virtual circuit disconnect 10.135.2.11:5064
epicsMutex pthread_mutex_lock failed: error Invalid argument

Jan 07 16:29:33 !!! Errlog message received (message is above)
epicsMutexOsdLock
Jan 07 16:29:33 !!! Errlog message received (message is above)
Thread _main_ (0xf2f31c0) can't proceed, suspending.

Jan 07 16:29:33 !!! Errlog message received (message is above)


Can anyone help with this problem please?

Regards,

Dr Mark Bennett
Principal Controls Engineer
Australian Synchrotron
This message and any attachments may contain proprietary or confidential information. If you are not the intended recipient or you received the message in error, you must not use, copy or distribute the message. Please notify the sender immediately and destroy the original message. Thank you.

Replies:
RE: Gateway fails with "epicsMutex pthread_mutex_lock failed: error Invalid argument" Mark Bennett

Navigate by Date:
Prev: Re: EPICs base MingW32 compilation failed ! Andrew Johnson
Next: Re: EPICs base MingW32 compilation failed ! Jens Eden
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  <20092010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: EPICs base MingW32 compilation failed ! Andrew Johnson
Next: RE: Gateway fails with "epicsMutex pthread_mutex_lock failed: error Invalid argument" Mark Bennett
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  <20092010  2011  2012  2013  2014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 31 Jan 2014 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·