EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RE: Bayer vs RGB1 performance for Prosilica cameras
From: Mark Rivers <[email protected]>
To: Bruce Hill <[email protected]>
Cc: "Nelson, Silke" <[email protected]>, "[email protected]" <[email protected]>
Date: Tue, 18 Feb 2014 12:18:52 +0000
Hi Bruce,

Now that you have fixed the Linux network buffer size, I think your problems are simply due to limitations of the camera and the limitations of GigE bandwidth.

You said that in RGB1 mode a single camera used to max out at 7-8 fps, and now you get 17+fps.  That is good, because I looked at the specs for that camera, and the maximum frame rate is indeed 17 fps.  So a single camera is performing at spec.

The problem with running 2 cameras is the limitation of GigE bandwidth.  Your cameras are 1388 x 1038 pixels.  In RGB1 mode each pixel is 3 bytes.  If you run the camera at 17 fps then the bandwidth in MB/s is:

(1388. * 1038 * 3 * 17) / (1024 * 1024)  = 70.07

So it is using 70.1 MB/s of GigE bandwidth.  GigE's maximum theoretical bandwidth is 125 MB/s.  So if you try to run 2 cameras in RGB1 mode at 17 fps you would need 140.2 MB/s, which is significantly more than GigE is capable of.  The cameras do not fail gracefully under these conditions, so the throughput plummets to 1-2 fps.

Here are some suggested solutions.

- Use Bayer mode.  In Bayer mode the interpolation from the color masked pixels to RGB is not done on the camera, it must be done on the host.  This reduces the network bandwidth by a factor of 3, because only 1 byte is required per pixel.  areaDetector has a NDColorConvert plugin that will convert Bayer to RGB.   The tradeoff is network bandwidth vs host CPU cycles.  If you have multiple fast cores this may be the best solution.

- The areaDetector Prosilica driver has a record called PSByteRate, which corresponds to their StreamBytesPerSecond control.  This allows you to tell the camera the maximum amount of network bandwidth that it is allowed to use.  This will limit the number of fps that each camera can do, but it will prevent the drastic loss in throughput when multiple cameras compete for bandwidth.

- The areaDetector Prosilica driver supports TriggerMode=FixedRate.  This allows you to limit the fps that each camera is running.  For example, if you set TriggerMode=FixedRate and AcquirePeriod=0.2 then the camera will only collect 5 fps, even if the exposure time would allow it to go faster.  If you do that then you should be able to run 2 or perhaps even 3 cameras at 5 fps without saturating GigE.

Mark

________________________________________
From: Bruce Hill [[email protected]]
Sent: Tuesday, February 18, 2014 3:58 AM
To: Mark Rivers
Cc: Nelson, Silke; [email protected]
Subject: Re: Bayer vs RGB1 performance for Prosilica cameras

Hi Mark,
That was a good suggestion, as my rmem_default was set to 128k.

Unfortunately, that helps, but not enough.
I can run one camera in RGB1 at 17+fps,
but if 2 cameras are in RGB1 I can't get them to go faster than 2 fps.

I've only tried one host machine w/ 2 cameras so far, w/ the other 2 off,
but trying to go any faster gets into the same
problem w/ dropped packets and the fps goes down instead of up.

I also tried setting rmem_default to 2MB instead of 1MB but no significant
improvement.    rmem_max is 16MB.

If I put one camera into bayer mode, I can run both at 18fps, or I can run
both at 18fps.    I'll do some more testing tomorrow.

Regards,
- Bruce

P.S.  I'm running setuid root w/ jumbo buffers

On 2/17/2014 8:30 PM, Mark Rivers wrote:
> Hi Bruce,
>
> This really sounds a lot like the problem of the Linux Ethernet network buffers being too small. We've seen similar problems where one camera would work fine, but then when trying to run multiple cameras on the same interface there would be lots of dropped packets and the rates go way down.
>
> This is from a message I sent on 4/22/2013 to the folks at LCLS who were having similar problems:
>
> ***********************************************************************
> Folks,
>
> I think I have solved the problem of dropping frames on the Prosilica cameras on Linux. The solution is simply to increase the value of the parameters net.core.rmem_max and net.core.rmem_default to 1048576.
>
> First, see what the current value of the parameters are on your system:
>
>> sudo sysctl -a | grep net.core.rmem
> On my system the values were the following:
>
> net.core.rmem_max = 131071
> net.core.rmem_default = 126976
>
> So they are about 128 KB. I then increased them to 1MB as suggested by the Point Grey article:
>
> http://www.ptgrey.com/support/kb/index.asp?a=4&q=354
>
> Colorado:areaDetector/iocBoot/iocPS3>sudo sysctl -w net.core.rmem_max=1048576 net.core.rmem_default=1048576
>
> Colorado:areaDetector/iocBoot/iocPS3>sudo sysctl -a | grep net.core.rmem
> net.core.rmem_max = 1048576
> net.core.rmem_default = 1048576
>
> Once I made those changes I could run 3 cameras on our Linux system with 4 cores and a single GigE interface and lose no frames at 5 frames/sec with both JPEG and Statistics plugins running. At 10 frames/sec I lost only 5 frames/3000, or less than 0.2%. Under these conditions all 4 cores are 100% busy, and I was previously losing 50% of the frames. You were losing 15% of the frames.
>
> Please give this a try ASAP and let me know if it fixes it for you as well.
>
> Mark
> ***********************************************************************
>
> This is the link to a tech-talk message that also discusses this:
>
> http://www.aps.anl.gov/epics/tech-talk/2013/msg00787.php
>
> Mark
>
>
> ________________________________________
> From: Bruce Hill [[email protected]]
> Sent: Monday, February 17, 2014 8:07 PM
> To: Mark Rivers
> Cc: Nelson, Silke; [email protected]
> Subject: Bayer vs RGB1 performance for Prosilica cameras
>
> Hi Mark,
> I've been digging into a problem we've been having with some of
> our Prosilica cameras not being able to sustain a decent frame rate
> when read via areaDetector.
>
> Most of our gigE cameras work great, but I traced the problem to our
> color cameras, which are Manta G146C, 1388x1038 w/ 4MB image.
> Best case we're getting 7-8fps on these, and when more than one was
> in use on an experiment, we couldn't run them reliably at more than 1fps
> unless we put each on it's own local ethernet adapter. As long as each
> camera has it's own adapter, they can run 7-8fps. My initial theory was
> that we were getting network packet failures as the camera is sending
> all the data via UDP packets.
>
> By chance, one ended up in free run and Bayer mode, and I found it
> running nearly 18hz. I setup a test w/ 4 IOC's split across 2 RHEL5 hosts, each
> reading one Manta G146C cameras via our switch, and can run all 4 at 18hz.
> Setting one to RGB1 causes it to drop to a few fps, w/ lots of packet errors
> and dropped frames.
>
> I tried making the exe owned by root w/ setuid, and I can toggle either camera
> between Bayer and RGB1 and both maintain 17+fps, but if I put both in RGB1,
> I start getting tons of packet errors and the fps plummets on both cameras.
>
> Have you seen any issues w/ RGB1 vs Bayer on these cameras?
>
> Is that something that would be handled in the areaDetector code, or
> should I be contacting Allied Vision?
>
> Thanks!
> - Bruce
>



Replies:
Re: Bayer vs RGB1 performance for Prosilica cameras Bruce Hill
References:
Bayer vs RGB1 performance for Prosilica cameras Bruce Hill
RE: Bayer vs RGB1 performance for Prosilica cameras Mark Rivers
Re: Bayer vs RGB1 performance for Prosilica cameras Bruce Hill

Navigate by Date:
Prev: epics base with MinGW-w64 Wang Xiaoqiang
Next: RE: Some CSS BOY questions Arnold, Ned D.
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: Bayer vs RGB1 performance for Prosilica cameras Bruce Hill
Next: Re: Bayer vs RGB1 performance for Prosilica cameras Bruce Hill
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  <20142015  2016  2017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 17 Dec 2015 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·