Experimental Physics and
Industrial Control System

"Jeff Hill" <[email protected]> · Thu, 14 Oct 2010 10:40:04 -0600

Mark,

You might try also updating your network drivers. Does this box have
multiple network interfaces such as both wireless and also LAN? I have
definitely seen PCs develop strange network stalls that driver updates have
fixed.

Another possibility might be some time discontinuities on this PC. Maybe
there is an NFS daemon which is yanking your time around and disrupting the
scheduling of timers in the CA client library.

> How do we go about figuring out what is wrong?

An unfortunate feature of windows is that netstat does _not_ show the number
of bytes pending against the tcp circuits which can be very helpful when
trying to determine what side of the tcp circuit has troubles. Nevertheless,
netstat could be helpful here because it will isolate between a tcp circuit
stall and an issue with the ca client library's PV discovery mechanism.
Basically, if you see a tcp circuit between the client and the IOC then you
know that it isn't an issue with CA's PV discovery mechanism, and it is
probably instead caused by IP kernel/driver issues, tcp forwarding issues in
the client, or tcp forwarding issues in the server. 

If you suspect issues in the IOC then you can run casr at a high level from
the IOC's shell and look to see if there are bytes pending to go out over
tcp, or waiting in ca server's input communication buffer for request
processing. 

If you suspect issues on the client side then that may require a debugger,
but a quick check will be to see if the client side gui is still responsive.
There is also a show capability in the client library, but you may need to
the debugger to run it (if the gui doesn?t have a popup that runs it)

Jeff
______________________________________________________
Jeffrey O. Hill           Email        [email protected]
LANL MS H820              Voice        505 665 1831
Los Alamos NM 87545 USA   FAX          505 665 5107

Message content: TSPA

> -----Original Message-----
> From: Mark Rivers [mailto:[email protected]]
> Sent: Wednesday, October 13, 2010 10:15 AM
> To: tech-talk; Jeff Hill
> Cc: Antonio Lanzirotti
> Subject: EPICS CA problems
> 
> Folks,
> 
> We are having trouble with a Windows IOC at NSLS.  Here are the
> symptoms:
> 
> - The IOC is running fine
> 
> - The PC running the IOC has 2 local CA clients connected to the IOC,
> medm and IDL.  Occassionally (1-2 times per day) one of these clients
> loses its connection to the IOC.  Medm screens go white, IDL says it
> cannot find a PV, etc.  This happens when the client was running fine.
> It typically only happens to one or the other client, not to both.
> 
> - Restarting the client fixes the problem.
> 
> - The same 2 clients are running on another PC connected to the same
> IOC.  Those clients are always fine, they do not lose connection when a
> client on the PC with the IOC does.
> 
> - Looking at the resources on the Windows machine (CPU, virtual and
> physical memory usage) does not indicate any problems.
> 
> How do we go about figuring out what is wrong?
> 
> Thanks,
> Mark

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System