EPICS FW: R3.13.10 ca

Experimental Physics and Industrial Control System

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 <2006> 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024	Index	1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 <2006> 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024
<== Date ==>		<== Thread ==>

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 <2006> 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024

Index

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 <2006> 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 2022 2023 2024

<== Date ==>

<== Thread ==>

Subject:

FW: R3.13.10 ca_event problem

From:

"Jeff Hill" <[email protected]>

To:

<[email protected]>

Date:

Wed, 25 Jan 2006 08:44:47 -0700

Title: Message

Allan,

The "callbackRequest ring buffer full" message comes from the database function

callbackRequest(). This typically indicates that a device's interrupt production rate

exceeds the the record processing rate (due to CPU saturation). This is related to

a device/record that are configured "scanned on interrupt".

> task 36a5718 CA_event suspended

My best guess at this time is that there is object code

corruption resulting from a errant pointer, but that is only

a wild guess, based on very limited information.

To debug this, we of course need the stack trace from that CA event thread after

this instruction access exception occurs. Type "tt ( event thread's task id )".

> appeared to lose it’s connections to all it’s clients

This would happen if there is CPU saturation. Also watch out for MBUF starvation

occurring with newer versions of vxWorks (which dont dynamically expand the

MBUF pool should it becomes depleted).

Jeff

-----Original Message-----
From: Al Honey [mailto:[email protected]]
Sent: Tuesday, January 24, 2006 12:03 PM
To: [email protected]
Subject: R3.13.10 ca_event problem

Aloha

Finally, 8 years after deploying R3.13.0beta12, we at the Keck observatory are attempting to deploy R3.13.10.

During about 80 hours of testing we have had 3 events of the form “ring buffer overflow” within a CA_event task. On two of those occasions the Vxworks 5.3.1 system (pPC), on which the error occurred appeared to lose it’s connections to all it’s clients, even other EPICS empowered IOC systems. The events lasted for 7-12 minutes, after which most connections were re-established (at least one channel to another IOC did not reconnect!). On the third such event a CA_event task was suspended and the following was in the iocLog:

tdc2:1040 Fri Jan 20 13:42:39 2006 filename="../taskwd.c" line number=159

tdc2:1040 Fri Jan 20 13:42:39 2006 task 36a5718 CA_event suspended

tdc2:1040 Fri Jan 20 13:43:28 2006 interrupt: ./taskwd.c" line number=159

tdc2:1040 Fri Jan 20 13:43:28 2006 callbackRequest ring buffer full

It so happened that I was monitoring the IOC via a telnet connection, thereby logging the following, which was not in the iocLog:

instruction access

Exception next instruction address: 0x30303030

Machine Status Register: 0x4000b030

Condition Register: 0x42000040

Task: 0x36a5718 "CA_event"^G

filename="../taskwd.c" line number=159

task 36a5718 CA_event suspended

interrupt: callbackRequest ring buffer full

Seeing as the instruction address 0x30303030 is outside of the address space… life does not seem too good.

I was not physically present during this event so I did not get a chance to perform a task trace before the IOC was rebooted.

I really hate to think that there is an errant pointer somewhere.

Any thoughts would be appreciated.

Thanks,

Al Honey

Kevin Tsubota