Experimental Physics and
Industrial Control System

Mark Rivers <[email protected]> · Tue, 18 Mar 2014 23:19:56 +0000

Hi Bruce,

Here is an update on sending Bayer images from the camera to the IOC, and doing the Bayer to RGB conversion in the IOC.

In current releases of areaDetector that Bayer conversion is done in the NDPluginColorConvert plugin.  The code uses the AVT PvApi library.  This means that the plugin depends on code from AVT, which is a dependency I would like to avoid.

For areaDetector R2-0, which will be released very soon, I have changed this.  The NDPluginColorConvert no longer does Bayer conversion.  Rather it is now done in the camera driver.  The Prosilica driver and the new Point Grey driver both now support this.  They each use the code from the respective vendor's API and library.

On a Point Grey camera I just bought (Black Fly BFLY-PGE-20E4C-CS) the performance gain is quite dramatic.  This is a 1600 x 1200 camera.  In RGB mode it does 14 frames/sec.  However, in Bayer mode it does 43.5 frames/s.  Both of those modes are about 80 MB/s, or about 2/3 of the theoretical GigE bandwidth.  At full frame rate with Bayer conversion on the host the CPU usage (with only the NDStdArrays plugin running) is less than 1%, which is pretty remarkable.  That camera is tiny (1 inch cube), uses Power Over Ethernet, and costs $600.

On my Prosilica 1380x1024 camera the frame rate increase is only from about 26 frames/s to 30 frames/s when going from RGB to Bayer.  The network bandwidth reduction is a factor of 3 at a given frame rate of course.  The CPU load with their library is also reasonable, perhaps 20% of a core.

Mark

________________________________________
From: Bruce Hill [[email protected]]
Sent: Tuesday, February 18, 2014 5:16 PM
To: Mark Rivers
Cc: Nelson, Silke; [email protected]
Subject: Re: Bayer vs RGB1 performance for Prosilica cameras

Hi Mark,
Excellent suggestions!
We've been using FixedRate to get by, but I think using
Bayer for the network capture, together w/ the byte rate limiting
should make for a much more stable solution.

Some spot checks w/ the color converter plugin look good
w/ 17fps through to the StdArray plugin.    I'll do some more tests
once I get them all setup this way.

Thanks and regards,
- Bruce

On 02/18/2014 04:18 AM, Mark Rivers wrote:
> Hi Bruce,
>
> Now that you have fixed the Linux network buffer size, I think your problems are simply due to limitations of the camera and the limitations of GigE bandwidth.
>
> You said that in RGB1 mode a single camera used to max out at 7-8 fps, and now you get 17+fps.  That is good, because I looked at the specs for that camera, and the maximum frame rate is indeed 17 fps.  So a single camera is performing at spec.
>
> The problem with running 2 cameras is the limitation of GigE bandwidth.  Your cameras are 1388 x 1038 pixels.  In RGB1 mode each pixel is 3 bytes.  If you run the camera at 17 fps then the bandwidth in MB/s is:
>
> (1388. * 1038 * 3 * 17) / (1024 * 1024)  = 70.07
>
> So it is using 70.1 MB/s of GigE bandwidth.  GigE's maximum theoretical bandwidth is 125 MB/s.  So if you try to run 2 cameras in RGB1 mode at 17 fps you would need 140.2 MB/s, which is significantly more than GigE is capable of.  The cameras do not fail gracefully under these conditions, so the throughput plummets to 1-2 fps.
>
> Here are some suggested solutions.
>
> - Use Bayer mode.  In Bayer mode the interpolation from the color masked pixels to RGB is not done on the camera, it must be done on the host.  This reduces the network bandwidth by a factor of 3, because only 1 byte is required per pixel.  areaDetector has a NDColorConvert plugin that will convert Bayer to RGB.   The tradeoff is network bandwidth vs host CPU cycles.  If you have multiple fast cores this may be the best solution.
>
> - The areaDetector Prosilica driver has a record called PSByteRate, which corresponds to their StreamBytesPerSecond control.  This allows you to tell the camera the maximum amount of network bandwidth that it is allowed to use.  This will limit the number of fps that each camera can do, but it will prevent the drastic loss in throughput when multiple cameras compete for bandwidth.
>
> - The areaDetector Prosilica driver supports TriggerMode=FixedRate.  This allows you to limit the fps that each camera is running.  For example, if you set TriggerMode=FixedRate and AcquirePeriod=0.2 then the camera will only collect 5 fps, even if the exposure time would allow it to go faster.  If you do that then you should be able to run 2 or perhaps even 3 cameras at 5 fps without saturating GigE.
>
> Mark
>
> ________________________________________
> From: Bruce Hill [[email protected]]
> Sent: Tuesday, February 18, 2014 3:58 AM
> To: Mark Rivers
> Cc: Nelson, Silke; [email protected]
> Subject: Re: Bayer vs RGB1 performance for Prosilica cameras
>
> Hi Mark,
> That was a good suggestion, as my rmem_default was set to 128k.
>
> Unfortunately, that helps, but not enough.
> I can run one camera in RGB1 at 17+fps,
> but if 2 cameras are in RGB1 I can't get them to go faster than 2 fps.
>
> I've only tried one host machine w/ 2 cameras so far, w/ the other 2 off,
> but trying to go any faster gets into the same
> problem w/ dropped packets and the fps goes down instead of up.
>
> I also tried setting rmem_default to 2MB instead of 1MB but no significant
> improvement.    rmem_max is 16MB.
>
> If I put one camera into bayer mode, I can run both at 18fps, or I can run
> both at 18fps.    I'll do some more testing tomorrow.
>
> Regards,
> - Bruce
>
> P.S.  I'm running setuid root w/ jumbo buffers
>
> On 2/17/2014 8:30 PM, Mark Rivers wrote:
>> Hi Bruce,
>>
>> This really sounds a lot like the problem of the Linux Ethernet network buffers being too small. We've seen similar problems where one camera would work fine, but then when trying to run multiple cameras on the same interface there would be lots of dropped packets and the rates go way down.
>>
>> This is from a message I sent on 4/22/2013 to the folks at LCLS who were having similar problems:
>>
>> ***********************************************************************
>> Folks,
>>
>> I think I have solved the problem of dropping frames on the Prosilica cameras on Linux. The solution is simply to increase the value of the parameters net.core.rmem_max and net.core.rmem_default to 1048576.
>>
>> First, see what the current value of the parameters are on your system:
>>
>>> sudo sysctl -a | grep net.core.rmem
>> On my system the values were the following:
>>
>> net.core.rmem_max = 131071
>> net.core.rmem_default = 126976
>>
>> So they are about 128 KB. I then increased them to 1MB as suggested by the Point Grey article:
>>
>> http://www.ptgrey.com/support/kb/index.asp?a=4&q=354
>>
>> Colorado:areaDetector/iocBoot/iocPS3>sudo sysctl -w net.core.rmem_max=1048576 net.core.rmem_default=1048576
>>
>> Colorado:areaDetector/iocBoot/iocPS3>sudo sysctl -a | grep net.core.rmem
>> net.core.rmem_max = 1048576
>> net.core.rmem_default = 1048576
>>
>> Once I made those changes I could run 3 cameras on our Linux system with 4 cores and a single GigE interface and lose no frames at 5 frames/sec with both JPEG and Statistics plugins running. At 10 frames/sec I lost only 5 frames/3000, or less than 0.2%. Under these conditions all 4 cores are 100% busy, and I was previously losing 50% of the frames. You were losing 15% of the frames.
>>
>> Please give this a try ASAP and let me know if it fixes it for you as well.
>>
>> Mark
>> ***********************************************************************
>>
>> This is the link to a tech-talk message that also discusses this:
>>
>> http://www.aps.anl.gov/epics/tech-talk/2013/msg00787.php
>>
>> Mark
>>
>>
>> ________________________________________
>> From: Bruce Hill [[email protected]]
>> Sent: Monday, February 17, 2014 8:07 PM
>> To: Mark Rivers
>> Cc: Nelson, Silke; [email protected]
>> Subject: Bayer vs RGB1 performance for Prosilica cameras
>>
>> Hi Mark,
>> I've been digging into a problem we've been having with some of
>> our Prosilica cameras not being able to sustain a decent frame rate
>> when read via areaDetector.
>>
>> Most of our gigE cameras work great, but I traced the problem to our
>> color cameras, which are Manta G146C, 1388x1038 w/ 4MB image.
>> Best case we're getting 7-8fps on these, and when more than one was
>> in use on an experiment, we couldn't run them reliably at more than 1fps
>> unless we put each on it's own local ethernet adapter. As long as each
>> camera has it's own adapter, they can run 7-8fps. My initial theory was
>> that we were getting network packet failures as the camera is sending
>> all the data via UDP packets.
>>
>> By chance, one ended up in free run and Bayer mode, and I found it
>> running nearly 18hz. I setup a test w/ 4 IOC's split across 2 RHEL5 hosts, each
>> reading one Manta G146C cameras via our switch, and can run all 4 at 18hz.
>> Setting one to RGB1 causes it to drop to a few fps, w/ lots of packet errors
>> and dropped frames.
>>
>> I tried making the exe owned by root w/ setuid, and I can toggle either camera
>> between Bayer and RGB1 and both maintain 17+fps, but if I put both in RGB1,
>> I start getting tons of packet errors and the fps plummets on both cameras.
>>
>> Have you seen any issues w/ RGB1 vs Bayer on these cameras?
>>
>> Is that something that would be handled in the areaDetector code, or
>> should I be contacting Allied Vision?
>>
>> Thanks!
>> - Bruce
>>

--
Bruce Hill
Member Technical Staff
SLAC National Accelerator Lab
2575 Sand Hill Road M/S 10
Menlo Park, CA  94025

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System