Experimental Physics and
Industrial Control System

"Jeff Hill" <[email protected]> · Mon, 29 Nov 2010 09:13:13 -0700

Hi Ben,

We have three things in some dependent threads here.
1) The udp part of the ca server is seeing protocol that isn't ca protocol from localhost? With typical routing configuration a ca client that broadcasts will be seen by ca servers that share the same host as the ca client, but on an IOC the ca client always checks for a local channel first with dbNameToAddr so these locally broadcasted search requests from an IOC resident ca client are typically not responded to by the IOC resident ca server.
2) The "CAS-event" thread, whose job is to read subscription updates from the ca event queue and send subscription updates to the tcp socket, has crashed.
3) The ei0 network interface was reset probably because an ip kernel watchdog decided that it has failed.

If one of these threads is at fault it has probably corrupted the data structures of the other two also.

The next time it happens please send a stack trace (using the "tt <task id>" command" for the " Task: 0x838cd8 "CAS-event" " thread that crashes. With power pc it isn't always easy to decipher the stack traces produced by the vxWorks shell's online disassembler. Better results can be expected from the tornado debugger, especially if itâs a debug build. Note that debug -g and optimized -O options are not mutually exclusive with gcc.

I would also check with wrs to see if ei0 driver patches are available.

Thanks,

Jeff
______________________________________________________
Jeffrey O. Hill           Email        [email protected]
LANL MS H820              Voice        505 665 1831
Los Alamos NM 87545 USA   FAX          505 665 5107

Message content: TSPA

With sufficient thrust, pigs fly just fine. However, this is
not necessarily a good idea. It is hard to be sure where they
are going to land, and it could be dangerous sitting under them
as they fly overhead. -- RFC 1925

> -----Original Message-----
> From: [email protected] [mailto:tech-talk-
> [email protected]] On Behalf Of Benjamin Franksen
> Sent: Monday, November 29, 2010 7:11 AM
> To: [email protected]
> Subject: Strange IOC crash
> 
> Hi All
> 
> we have a strange IOC crash that today happened for the second time in
> about
> two weeks. This is what we see:
> 
> [Mon Nov 29 00:23:03 2010]../udpiiu.cpp: Undecipherable (payload too
> small)
> UDP msg from 127.0.0.1:1034 ignored
> [Mon Nov 29 00:23:03 2010]../udpiiu.cpp: Undecipherable (payload too
> small)
> UDP msg from 127.0.0.1:1036 ignored
> [Mon Nov 29 00:23:03 2010]
> [Mon Nov 29 00:23:03 2010]Access Fault
> [Mon Nov 29 00:23:03 2010]Program Counter: 0x0003d152
> [Mon Nov 29 00:23:03 2010]Status Register: 0x3004
> [Mon Nov 29 00:23:03 2010]Access Address : 0xcffea3c9
> [Mon Nov 29 00:23:03 2010]Special Status : 0x0145
> [Mon Nov 29 00:23:03 2010]Task: 0x838cd8 "CAS-event"
> [Mon Nov 29 00:23:05
> 2010]filename="../../../src/libCom/taskwd/taskwd.c"
> line number=174
> [Mon Nov 29 00:23:05 2010]task 0x838cd8 suspended
> [Mon Nov 29 00:23:12 2010]interrupt:
> [Mon Nov 29 00:23:12 2010]ei0: reset
> 
> There has been no activity on this IOC during the last 12 hours, except
> periodic scanning of status and current readbacks over CAN bus.
> 
> This is under VxWorks 5.4.2 on an mv2100 board, EPICS base R3.14.8.2.
> 
> What is most strange is that the bad UDP message supposedly comes from
> localhost.
> 
> Any kind of suggestion as to what could cause this would be greatly
> appreciated.
> 
> Cheers
> Ben

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System