Experimental Physics and
Industrial Control System

"Jeff Hill" <[email protected]> · Mon, 25 Oct 2010 10:13:59 -0600

Hi Pawel,

> A call to 'assert(status == epicsMutexLockOK)'
>      by thread 'CAS-client' failed in ../dbEvent.c line 460.

This happening when the ca server is switching out of flow control mode. It would be useful to type "tt <task id>" for the suspended thread, and look at the argument passed to db_event_enable. Dumping the memory for that argument (a pointer to a dbEventSubscription structure), using "d <address>", might reveal if this is memory corruption or some other type of bug. The corruption scenario _is_ somewhat more likely with a new device support that copies large blocks of memory around, but of course we can't rule out at this point the possibility of a new bug in the server that hasnât been seen before; the fact that flow control mode is active does point to some modal behaviors in the server that might not be used by all clients (less often used by clients on fast workstations). The server bug scenario might also possibly result from how your application switches banks of channels, if this was different from other clients. Does your client switch banks of data basically by des!
 troying CA channels and creating new ones. I suspect this is the case, and therefore maybe this isn't substantially different from other clients, and then perhaps a server bug is contraindicated, at least when considering that particular usage pattern in isolation.

The channel access system has had from the beginning a feature (we call channel access flow control) where the client will try to not allow a large time lag between what it updates on an operator screen and the subscription updates produced by the IOC. This requirement for this feature was coming from an old VMS GUI environment called UIS which was quite pathetically slow drawing text on its screen, and could actually become 15 minutes behind the data being produced in the IOC. In flow-control mode the server stops sending subscription updates, but retains always the latest value produced by the database. When exiting flow-control mode the server sends a subscription update to the client for any subscription which changed when flow control mode was active. The client switches into flow control mode when it has read many frames from the tcp circuit without a gap in time in-between any of them, and exits flow control mode when it has caught up again with current updates from t!
 he IOC.

Here is another fault isolation option. You could write a simulation version of your device support which does not access hardware, but attempts to reproduce similar memory transport and asynchronous behaviors, and run it on Linux in a soft IOC under valgrind to see if there are any out-of-bounds memory references.

Jeff
______________________________________________________
Jeffrey O. Hill           Email        [email protected]
LANL MS H820              Voice        505 665 1831
Los Alamos NM 87545 USA   FAX          505 665 5107

Message content: TSPA

> -----Original Message-----
> From: [email protected] [mailto:[email protected]]
> On Behalf Of Pawel Predki
> Sent: Monday, October 25, 2010 8:21 AM
> To: TechTalk
> Subject: CSS panel switching causes epicsMutexLockOK assert fail
> 
> Hello,
> 
> I'm having more problems with the data acquisition software I'm
> developing. This time I have no idea where the error I'm getting is
> coming from.
> 
> The idea is this - there are 12 DAQ cards in the system, each able to
> support 4 channels. There are 12 action buttons which enable the
> operator to switch between the DAQ cards and one XY Graph that displays
> the waveforms from all 4 channels for each card. The contents of the
> graph is controlled by a $(cardno) macro, which is changed whenever the
> action button is pressed. To be more precise, another panel is loaded
> with a different value of $(cardno) and it replaces the previous one.
> 
> It is possible to switch between the cards without any problems until
> data starts coming. I wrote my own device support for the cards using
> the waveform record. Data is sent over TCP/IP and once a full sample set
> is received a scanIoRequest function is called which, in turns, calls
> the read_waveform function where I fill the buffer with valid data.
> 
> As I said, once the data starts coming, switching from one card to
> another causes this error to appear:
> 
> A call to 'assert(status == epicsMutexLockOK)'
>      by thread 'CAS-client' failed in ../dbEvent.c line 460.
> EPICS Release EPICS R3.14.11 $R3-14-11$ $2009/08/28 18:47:36$.
> Local time is 2010-10-25 16:11:12.177758908 CEST
> Please E-mail this message to the author or to [email protected]
> Calling epicsThreadSuspendSelf()
> 
> A call to 'assert(status == epicsMutexLockOK)'
>      by thread 'CAS-client' failed in ../dbEvent.c line 442.
> EPICS Release EPICS R3.14.11 $R3-14-11$ $2009/08/28 18:47:36$.
> Local time is 2010-10-25 16:11:12.457952837 CEST
> Please E-mail this message to the author or to [email protected]
> Calling epicsThreadSuspendSelf()
> Thread CAS-client (0x82eb670) suspended
> Thread CAS-client (0x8308220) suspended
> 
> 
> I looked in dbEvent.c but couldn't find anything that would have
> anything to do with my problem. Does anyone have any ideas what could be
> wrong? What could hold the mutex and what is the mutex really guarding
> here?
> 
> Kind Regards,
> Pawel Predki

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System