Experimental Physics and
Industrial Control System

Mark Rivers <[email protected]> · Thu, 22 Aug 2013 17:31:11 +0000

Hi Zen,

The problem will only occur if the Posix signals are being generated in the same application (e.g. IOC) that is running the drvAsynIPPort driver.  So having an IOC running areaDetector would not cause the problem with the Beckhoff running in a different IOC on the same Linux machine.

However, the problem is not restricted to the Prosilica vendor library, areaDetector, or even the Posix high-resolution timer routines.  The problem can happen with any software that is using certain Posix signals.  Those signals will cause many system calls (e.g. poll(), write(), read(), send(), recv(), nanosleep(), etc.) to terminate early with errno=EINTR.  Any code that calls those functions must thus check the status return, and if it fails then check if errno=EINTR.  If errno=EINTR the operation may need to be retried.

Mark

-----Original Message-----
From: Zenon Szalata [mailto:[email protected]] 
Sent: Thursday, August 22, 2013 12:20 PM
To: Mark Rivers; [email protected]; Michael Dunning
Subject: Re: Comms issue with New Focus 8752 motor driver

Hi Mark,
This is very interesting.  Could it be that this effect is also 
responsible for my problems with Beckhoff devices?  I have made 
modifications to both the drvAsynIPPort to test the return code from the 
poll() routine a few weeks ago when I started suspecting this. 
Unfortunately, shortly before I made the modification, the Beckhoff IOC 
errors stopped when the laboratory went into summer shutdown mode.  I 
still don't understand what was causing the Beckhoff errors in the first 
place.  I take it from your description of the problem that if area 
detector is running on the same Linux machine where an IOC which is 
using drvAsynIPPort is, then the IOC might be affected by the area 
detector generating an EINTR signal.  I am not sure that this is exactly 
the situation here.  I need to check into this. Could some other EPICS 
IOC do similar thing as the area detector does?
Zen

On 08/22/13 08:20, [email protected] wrote:
> Following some off-line discussions with Mark, I can report the actual problem wasn't specific to the 8752 driver, but an interaction between the Prosilica AreaDetector camera driver (which my IOC was also running) and asyn which was causing the 8752 driver's asyn read call to return prematurely.  Mark described the problem much more eloquently than I can:
>
>> The Prosilica library uses Posix high-precision timer library.  That library uses signals that can cause Linux system functions to return
>> prematurely returning EINTR, so they must be restarted.  Andrew Johnson fixed this problem in base some time ago when I first discovered
>> the problem when using Prosilica cameras.
>>
>> I am pretty sure the drvAsynIPPort module has a bug.  It is calling poll(), but not checking the response at all.  It assumes that if poll() returned
>> then the port has new data.  But this is incorrect, because according to the documentation poll() can return EINTR
> Mark has now updated drvAsynIPPort.c in revision r2251 at https://svn.aps.anl.gov/epics/asyn/trunk/ which fixes the problem nicely, so doubtless it appear in the next asyn release also.
>
> Many thanks to Mark for the fix, and to Eric and Kevin for their helpful suggestions.
>
> Tim
>
> -----Original Message-----
> Date: Tue, 20 Aug 2013 19:25:19 +0000
> From: Mark Rivers <[email protected]>
> To: "'[email protected]'" <[email protected]>,
> 	"[email protected]"	<[email protected]>
> Subject: RE: Comms issue with New Focus 8752 motor driver
>
> It seems that sometimes when the VER command is issued the response does not arrive before the timeout expires.  But then it does eventually arrive, and screws up the expected response to the DRT command.  The timeout is defined on this line in the driver:
>
> #define PMNC_TIMEOUT    3.0     /* Command timeout in sec. */
>
> There is something I really don't understand in the output:
>
> 2013/08/20 17:37:57.709 192.168.42.31:23 write 4
> VER\r
> 2013/08/20 17:37:57.716 192.168.42.31:23 read 1
> send_recv_mess(): recv message = ""
> send_recv_mess(): message = VER
> 2013/08/20 17:37:57.717 192.168.42.31:23 write 4
> VER\r
> send_recv_mess(): ERROR - staus =1, nread = 0
> send_recv_mess(): message = VER
> 2013/08/20 17:37:57.736 192.168.42.31:23 write 4
> VER\r
> 2013/08/20 17:37:57.742 192.168.42.31:23 read 16
> Version 1.6.0\r\n>
> send_recv_mess(): recv message = "Version 1.6.0
> "
>
> The line
>
> send_recv_mess(): ERROR - staus =1, nread = 0
>
> Is coming from this code (note the typo on status, spelled staus):
>
>      /* Perform atomic write/read operation  */
>      status = pasynOctetSyncIO->writeRead(cntrl->pasynUser, send_com, size,
>                                           recv_com, BUFF_SIZE-1,
>                                           PMNC_TIMEOUT, &nwrite, &nread, &eomReason);
>      if (eos)
>        // Reset EOS to default
>        pasynOctetSyncIO->setInputEos(cntrl->pasynUser,PROMPT_EOS,strlen(PROMPT_EOS));
>
>      if (status != asynSuccess)
>      {
>        int int_status = (int) status;
>        Debug(3, "send_recv_mess(): ERROR - staus =%d, nread = %d\n", int_status, nread);
>        recv_com[0] = '\0';
>        return(MESS_ERR);
>      }
>
> status=1 is asynTimeout.  So this is saying that the writeRead operation failed with a timeout on the read, and nread=0.
>
> But look at the timestamps above. Only 19 ms has elapsed between the second 2 "write 4" messages, but in that time the read has timed out!  This does not make sense, there should be a 3 second delay in there before the read times out.
>
> Mark
>

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System