EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: Area Detector
From: Zenon Szalata <[email protected]>
To: Mark Rivers <[email protected]>
Cc: tech-talk <[email protected]>
Date: Sat, 9 Mar 2013 20:26:27 -0800
Mark, thanks for the explanation.
I calculated the data rate and realized that it was needed to reduce the image size. I did set the to 2 in both X and Y. So, the rates we were dealing with was 1/4 of the rate you calculated. Unfortunately, I forgot to mention this in my previous emails. On Monday, I will adjust the PSByteRate accordingly and do some more testing. Actually, the operators did not get a chance yet to use the camera images. I just installed the camera last Friday and I was testing using the camera IOC from the control room with a colleague of mine.
Zen

On 03/09/13 19:56, Mark Rivers wrote:
Hi Zen,

Looking at the original error messages:

2013/03/08 12:20:26.833 prosilica:readStats: error, status=14
2013/03/08 12:20:26.833 prosilica:readParameters: error, status=8
2013/03/08 12:20:26.833 prosilica:writeInt32: error, status=8
readParameters is returning error 8, and readStats is returning 14.  These functions OR together the errors from any Prosilica API functions they call.  The error definitions in PvAPI.h are:

     ePvErrNotFound      = 6,        // Camera or attribute not found
     ePvErrAccessDenied  = 7,        // Camera cannot be opened in the specified mode
     ePvErrUnplugged     = 8,        // Camera was unplugged
This strongly suggests to me that the vendor library is returning 8 (camera was unplugged) for some functions, and 6 (camera or attribute not found) for other functions, since 14 is the OR of 8 and 6.

I suspect what is happening is that the IOC is losing connection with the camera when you are starting ImageJ from the control room.  I can't really explain why this happens when ImageJ is displaying in the control room but not in your office, since that should only be related to the network connection between the IOC and the camera.  It should not depend on where ImageJ is sending its images across the network.  Are you sure they are using the same parameters (frame rate, binning, etc.) when they use it from the control room?

Since you are running on 100Mbit Ethernet it is very important that you set the PSByteRate PV correctly.  It is documented here:

http://cars9.uchicago.edu/software/epics/prosilicaDoc.html

This PV corresponds to the StreamBytesPerSecond parameter in the vendor documentation. It limits the bandwidth that the camera and the vendor library will try to use.  On a 100 Mbit network you should set this to 10000000 (10 million) or less, because that is all that 100 Mbit Ethernet can support.  You might want to set it even a bit lower.  If that is not set to a valid value it could be causing the camera to timeout and disconnect.

You said that the camera resolution is 1292x964, and you were running it at 10 frames/sec.  We can then calculate the bit rate:

(1292 * 964) pixels/frame * 10 frames/sec * 8 bits/pixel = 99639040 bits/sec

So the bit rate is 99.6 Mbits/sec.  This is almost exactly the theoretical bit rate of 100Mbit Ethernet!  So I am not surprised it is having problems.  It is probably right on the hairy edge of working.  You should reduce the frame rate to 8 frames/sec maximum, and set PSByteRate to 10000000.

Note that in areaDetector R1-8 and earlier the Prosilica driver will not reconnect to the camera if the camera disconnects and then reconnects.  In R1-9 autoconnection and disconnect/reconnect is implemented (thank to Jason Abernathy from University of Victoria).  So if you upgrade to R1-9 it may allow you to reconnect to the camera even if the connection is lost.

Mark

________________________________________
From: Zenon Szalata [[email protected]]
Sent: Saturday, March 09, 2013 1:24 PM
To: Mark Rivers
Cc: tech-talk; Dunning, Michael; Nelson, Janice L.
Subject: Re: Area Detector

Hi Mark,
I will not be able to do the tests you suggest until Monday.  I can add
mode details to the picture I have painted so far.
Subnet SA is in our experimental hall, where we have a 100Mbit switch (I
know that it is not suitable for handling camera images, but that is all
we can presently afford).  This switch is configured for two subnets, SA
and SAr, where SAr is a restricted subnet, which is only visible from
SA.  This is done to isolate devices like cameras, Beckhoff, and other
non computer network devices so that they do not get clobbered by
security scans.  The camera is on SAr. Computer CA is on SA.  There is
no problem when I ssh from my desktop Linux computer, which is on one of
SLAC public subnets, to CA, run ImageJ on CA and I can see images at
more than 10Hz.

I don't really know the details of the rest of the network path from CC
to CA.  That is, I don't know how many switches and their speed are in
the path.  I suspect that these switches are probably 1 GBit.

Subnet SC, where computer CC is used by the accellerator operator, is a
private subnet.
I think that there is a dual hosted computer, which is most likely CB,
that lives on both subnet SC and SB.  I suspect that the problem is
really in going from computer CC to CB.

I have performed the following test:
  From my desktop computer I did ssh to computer CB and from there I did
ssh to CA.  Then still working from my desktop computer I started ImageJ
on CA and I was geting images at 10Hz without any errors.
I am thinking naively that a limited networking bandwidth should slow
down the rate at which images are moved but should not put the IOC in an
unrecoverable bad state.  But I know that naive thinking is usually wrong.
The networking expert in charge of SC and SB was away this week and
should be back at work next week and we will discuss this with him.
Thanks for any suggestions,
Zen
On 03/09/13 10:07, Mark Rivers wrote:
Zen,

3 questions:

1) Is the camera on subnet SA, the same subnet the IOC is running on?

2) Is ImageJ running on CA, i.e. the same computer as the IOC?

3) Are you sure that the entire path from CC to CA, and from CA to the camera is Gigabit, and there are no 100Mbit switches or hubs in the path?

Here is something to try.  Throttle the NDPluginStdArrays plugin, which is the plugin that converts images to waveform records.  Set the MinCallbackTime PV to 1.0 so it only updates the waveform record at 1Hz, or 0.5 second to limit it to 2Hz.  This will do 2 things:

- Reduce the CPU load on CA, assuming that ImageJ is also running on CA, because ImageJ will now only try to display 1 frame/sec.
- Reduce the network bandwidth used between CA and CC.  If you are running ImageJ on CA then you are sending images over X11 between CA and CC.

You can then run "top" on CA and see what the CPU load is as you reduce MinCallbackTime.  As it begins to fail can see if CPU load is the problem or not.

Mark

________________________________________
From: Zenon Szalata [[email protected]]
Sent: Saturday, March 09, 2013 11:07 AM
To: Mark Rivers; tech-talk; Dunning, Michael; Nelson, Janice L.
Subject: Area Detector

Hi Mark,
I have a peculiar problem with an IOC using an area detector.  It
controls a prosilica camera (not sure what is the model, it has 1292x964
image size, monochrome).  Area detector version is 1.8, asyn 4.20, and
EPICS 3.14.12.2,
It all works fine.
A problem is encounred when a few hops are needed to get to the computer
where the soft IOC runs.
This is how we are doing it:
there are three subnets, call them SA, SB, and SC.  Three computers are
involved:
the operator is sitting in front of computer CC on subnet SC.  The
operator does ssh from CC to computer CB on subnet SB, and from there
does ssh to computer CA on subnet SA.  The IOC runs on CA. This is
needed because the accelarator operators can only get to subnet SA as
described above via two hops.  The prosilica EDM control screens work
fine and there is no problem until we launch ImageJ.  It connects and
the IOC starts printing the following messages:
2013/03/08 12:20:21.833 PS1:cam1:PSReadStatistics devAsynInt32 process
error
2013/03/08 12:20:26.833 prosilica:readStats: error, status=14
2013/03/08 12:20:26.833 prosilica:readParameters: error, status=8
2013/03/08 12:20:26.833 prosilica:writeInt32: error, status=8
function=76, value=0
Stopping ImageJ does not clear the problem.  It seems that the only way
to get out of this is to restart the IOC.
Strangely, the EDM viewer widget does somewhat better.  We can use one
of those, but the same problem accurrs when a second EDM viewer is started.
It seems to be a bandwidth problem.  But why does the IOC get into the
mode where it cant process periodic chores and won't recover?
Any ideas or suggestions would be very helpful.
Thanks,
Zen


References:
Area Detector Zenon Szalata
RE: Area Detector Mark Rivers
Re: Area Detector Zenon Szalata
RE: Area Detector Mark Rivers

Navigate by Date:
Prev: RE: Area Detector Mark Rivers
Next: Re: Area Detector Zenon Szalata
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: RE: Area Detector Mark Rivers
Next: Re: Area Detector Zenon Szalata
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  <20132014  2015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 20 Apr 2015 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·