Experimental Physics and
Industrial Control System

"Mark Rivers" <[email protected]> · Mon, 25 Oct 2010 11:55:27 -0500

Hi Jeff,

I have some more information on this.  The problem does NOT appear to be a problem with caRepeater crashing.  When the client loses connection to the IOC the Windows Task Manager shows that caRepeater is still running on the IOC PC.  Normally we had been seeing the problem when the client and the IOC were running on the same computer.

However, last night we managed to reproduce the problem with the client running on a separate PC.

I have attached the output of casr(100) on the IOC when the client has lost communication.  The IOC server is 172.16.1.20 (X26A-Control) and the client is running on 172.16.1.21 (X26A-Data).

It appears that when this happens the client loses connection to all PVs on the server.  But we know for sure that it lost connection to X26A:med:Acquiring.

I think I see something suspicious in the output.  Here is the start of one block of output from casr for the client machine that has lost connection:

TCP 172.16.1.21:1726(X26A-Data): User="X26A User", V4.11, 1755 Channels, Priority=0
        Task Id=0x12a022b0, Socket FD=15
        Secs since last send   0.02, Secs since last receive   0.02
        Unprocessed request bytes=0, Undelivered response bytes=0
        State=up
        360696 bytes allocated
        X26A:med:PresetMode(0rw)        X26A:med:ElapsedReal(1rw)       X26A:med:PresetReal(0rw)

Here is the start of another block for the same client:

TCP 172.16.1.21:1752(X26A-Data): User="X26A User", V4.11, 1837 Channels, Priority=0
        Task Id=0x12ec19d0, Socket FD=16
        Secs since last send 131528.55, Secs since last receive 140881.60
        Unprocessed request bytes=8296, Undelivered response bytes=0
        State=up
        377424 bytes allocated

Note that there are unprocessed request bytes there.

There is then another block for the same client machine:

TCP 172.16.1.21:2735(X26A-Data): User="X26A User", V4.11, 1837 Channels, Priority=0
        Task Id=0x12ec0a10, Socket FD=19
        Secs since last send 119.81, Secs since last receive 119.82
        Unprocessed request bytes=0, Undelivered response bytes=0
        State=up
        377424 bytes allocated

There is also a UDP entry for that client machine:

UDP Server:
UDP 172.16.1.21:2733(): User="", V4.11, 0 Channels, Priority=0
        Task Id=0x1293c4e0, Socket FD=11
        Secs since last send 131525.68, Secs since last receive   3.06
        Unprocessed request bytes=16, Undelivered response bytes=0
        State=up
        180 bytes allocated

        Send Lock

I am not sure how to interpret this.

Thanks,
Mark

-----Original Message-----
From: Jeff Hill [mailto:[email protected]] 
Sent: Tuesday, October 19, 2010 10:33 AM
To: Mark Rivers; 'tech-talk'
Cc: Antonio Lanzirotti
Subject: RE: EPICS CA problems

Hi Mark,

This is the first I have heard of any issues the ca repeater crashing.

Is this running under cygwin or mingw? Compiled by ms visual c or gnu?

The stack trace has no symbols so it's hard to determine a cause. If you
could fire up the relevant debugger and get a stack trace with symbols that
would help. You might need to build base for debugging. Set HOST_OPT=YES in
CONFIG_SITE. Also, if you save the debugging session in visual c++ and email
it to me I might be able to identify the issue.

Jeff
______________________________________________________
Jeffrey O. Hill           Email        [email protected]
LANL MS H820              Voice        505 665 1831
Los Alamos NM 87545 USA   FAX          505 665 5107

Message content: TSPA

> -----Original Message-----
> From: Mark Rivers [mailto:[email protected]]
> Sent: Monday, October 18, 2010 7:58 PM
> To: tech-talk; Jeff Hill
> Cc: Antonio Lanzirotti
> Subject: RE: EPICS CA problems
> 
> Folks,
> 
> I learned today that it appears that caRepeater has been crashing on
> this system.  I don't know for sure that this problem happens when
> caRepeater has died, but that seems likely.  The next time it happens
> we will look to see if caRepeater is still running.
> 
> Meanwhile, we have found that there are caRepeater stackdump files,
> containing the following:
> 
> Exception: STATUS_ACCESS_VIOLATION at eip=610B9F69
> eax=00000000 ebx=00000001 ecx=00000000 edx=0014C6F0 esi=00000000
> edi=011DCCD8
> ebp=011DCB14 esp=011DCAEC program=C:\Program Files\EPICS WIN32
> Extensions\caRepeater.exe, pid 2152, thread unknown (0xC44)
> cs=001B ds=0023 es=0023 fs=003B gs=0000 ss=0023
> Stack trace:
> Frame     Function  Args
> 011DCB14  610B9F69  (00000000, 00000000, 00000000, 00000000)
> 011DCC24  610BA905  (00000000, 00000000, 00000000, 00000000)
> 011DCCE4  610BB67A  (FFFFFFFF, FFFFFFFF, 00000000, 00000000)
> 011DCD34  61027DE2  (00000002, 011DCE64, 00000002, 011DCE00)
> 011DCDC8  7C87655C  (00000002, 011DCE00, 7C8763C0, 00000002)
> End of stack trace
> 
> Has anyone else seen such stackdumps from caRepeater?  This is the
> version of caRepeater.exe that is included in the most recent (Nov. 2,
> 2007) APS "EPICS Win32 Extensions" package.
> 
> Thanks,
> Mark
> 
> 
> ________________________________
> 
> From: Mark Rivers
> Sent: Wed 10/13/2010 11:14 AM
> To: tech-talk; 'Jeff Hill'
> Cc: Antonio Lanzirotti
> Subject: EPICS CA problems
> 
> 
> 
> Folks,
> 
> We are having trouble with a Windows IOC at NSLS.  Here are the
> symptoms:
> 
> - The IOC is running fine
> 
> - The PC running the IOC has 2 local CA clients connected to the IOC,
> medm and IDL.  Occassionally (1-2 times per day) one of these clients
> loses its connection to the IOC.  Medm screens go white, IDL says it
> cannot find a PV, etc.  This happens when the client was running fine.
> It typically only happens to one or the other client, not to both.
> 
> - Restarting the client fixes the problem.
> 
> - The same 2 clients are running on another PC connected to the same
> IOC.  Those clients are always fine, they do not lose connection when a
> client on the PC with the IOC does.
> 
> - Looking at the resources on the Windows machine (CPU, virtual and
> physical memory usage) does not indicate any problems.
> 
> How do we go about figuring out what is wrong?
> 
> Thanks,
> Mark
> 

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System