Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  <2017 Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  <2017
<== Date ==> <== Thread ==>

Subject: RE: Area Detector and high performance NVME devices
From: "Mark S. Engbretson" <Engbretson@anl.gov>
To: "'Mark Rivers'" <rivers@cars.uchicago.edu>, <tech-talk@aps.anl.gov>
Date: Tue, 27 Jun 2017 15:07:35 -0500
Pete suggested something like that,  have the HDF file have a pointer to an
each image raw file, but can not sustain the write rate with single files.
Why I was originally trying to tweak this file writer into something that
would acquire all the data, but would then generate reasonable output after
the fact.

2BM seems to be willing to have a starting point of perhaps one 15 minute
acquire per hour, in which the  other 45 minutes either processing the data
or getting it off the computer to prepare for another. Which may just be
enough if the raw data is buffered and HDF write is out at whatever rate
that it can keep up at.

I hadn't thought of this raw plugin abstracting a much smaller image that
could be a placeholder in an HDF file. Mostly since someone would still have
to post process both of these files after the fact. No file plugins have
implemented the read funcstions yet.

-----Original Message-----
From: Mark Rivers [mailto:rivers@cars.uchicago.edu] 
Sent: Tuesday, June 27, 2017 2:48 PM
To: Mark S. Engbretson <Engbretson@anl.gov>; tech-talk@aps.anl.gov
Subject: RE: Area Detector and high performance NVME devices

Hi Mark,

I just tried with simDetector on a Windows 7 system with 8 cores, 15K RPM
SAS Raid-0 disk, 96 GB RAM.

4096 x 3078 Int8 images = 12 MB/image.

The simDetector is generating about 150 frames/s = 1.8 GB/s.

This is the output of camonitor on the ArrayRate_RBV and WriteFile_RBV PVs
in the HDF5 plugin:

corvette:simDetectorIOC/iocBoot/iocSimDetector>camonitor -tc
13SIM1:HDF1:ArrayRate_RBV 13SIM1:HDF1:WriteFile_RBV
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:21.809029) 0
13SIM1:HDF1:WriteFile_RBV      (2017-06-27 14:15:21.809162) Done
13SIM1:HDF1:WriteFile_RBV      (2017-06-27 14:15:31.451841) Writing STATE
MINOR
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:33.410624) 34
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:34.411782) 54
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:35.410801) 55
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:37.408966) 52
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:38.408085) 54
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:39.407156) 48
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:40.408254) 53
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:41.409449) 55
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:42.410515) 45
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:43.411498) 50
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:44.411601) 47
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:45.410666) 53
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:46.409790) 50
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:47.408919) 54
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:48.407977) 53
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:49.407138) 52
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:50.408088) 60
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:51.409265) 62
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:52.410457) 52
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:53.411547) 59
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:54.411524) 62
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:55.411724) 55
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:56.410814) 60
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:57.409946) 57
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:58.408960) 59
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:15:59.407069) 60
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:00.408095) 42
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:01.409358) 17
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:02.410576) 27
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:03.411677) 25
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:04.413487) 27
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:05.411605) 25
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:06.411612) 24
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:07.409803) 23
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:11.410220) 19
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:12.411320) 18
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:13.412261) 21
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:15.412525) 20
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:16.411553) 22
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:17.410808) 24
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:20.409031) 28
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:21.411129) 25
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:22.411199) 24
13SIM1:HDF1:WriteFile_RBV      (2017-06-27 14:16:22.818068) Done
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:23.412388) 1
13SIM1:HDF1:ArrayRate_RBV      (2017-06-27 14:16:24.413361) 0

So for the first 25 seconds or so it is writing at about 55 frames/s = 660
MB/s.  This is probably filling the Windows file cache.  It then slows down
to about 23 frames/s = 280 MB/s, which is probably the steady state write
speed of the disks.

So I agree that it will probably be difficult to write HDF5 files at the
full rate of your camera, which is 190 frame/s = 2.3 GB/s.

One possible solution would be to write your RAW files that can keep up, and
also write HDF5 files of "thumbnail" data that is either cropped or binned
to 512x384 for example.  You can then store all the metadata in the HDF file
and the images in the RAW file.  Later on you can either merge these 2 files
into a large HDF5 file, or just keep them separate.

Mark

________________________________
From: Mark S. Engbretson [Engbretson@anl.gov]
Sent: Tuesday, June 27, 2017 1:51 PM
To: Mark Rivers; tech-talk@aps.anl.gov
Subject: RE: Area Detector and high performance NVME devices

I do not have enough memory to create a queue large enough to buffer all the
images.  2BM wants to acquire the camera stream for at least 15 minutes and
ideally as long as possible. So talking about 2-4 TB.  Or larger if the buy
huge nvme chips or multiple turbo Z units - the computer supports having 3
of them.

I have allocated very large buffers, but for a 4096 by 3078 image being
created at 190 FPS, the file plugins would have to be able to keep up . . .
and they don't.

Using the SimDetector, I can create such images at ~260 FPS. HDF is only
writing the file out at about 60 FPS, using whatever the defaults settings.
NetCFD writes about 30 FPS.  A buffer of 4000 in both cases only lasted for
about a minute. The raw file plugin slows the simDetector acquire rate to
about 170-180 FPS, which the plugin can keep up with. The stardardarray
plugin by itself also slows simDetecor to about  the same thing.


From: Mark Rivers [mailto:rivers@cars.uchicago.edu]
Sent: Tuesday, June 27, 2017 1:06 PM
To: 'Mark S. Engbretson' <Engbretson@anl.gov>; tech-talk@aps.anl.gov
Subject: RE: Area Detector and high performance NVME devices

Hi Mark,

I am surprised that a raw file plugin is significantly faster than netCDF or
HDF5.  I would like to see the tests, and figure out what is actually
slowing them down, i.e. is it CPU bound, waiting for a semaphore, etc.?  Can
you post actual benchmark results for the different plugins, i.e. frames/s
and MB/s?

You should not need to do anything special to create a FIFO to buffer images
while the disk is busy.  Every areaDetector plugin comes with such a FIFO,
i.e. its input queue.  Just increase the QueueSize to be large enough to
buffer all the images you need to store in one "burst".  You can also use
the CircularBuffer plugin to do this, but it should really not be necessary,
that is intended more for "triggered" applications where the buffer is
emptied when a trigger condition is satisfied.

Mark


From: Mark S. Engbretson [mailto:Engbretson@anl.gov]
Sent: Tuesday, June 27, 2017 12:24 PM
To: Mark Rivers; tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov>
Subject: Area Detector and high performance NVME devices

Mark -

I have the adimec camera which generates data at  ~2.5 GB/s. I recently got
my hands on a newer HP 840 with a HP Turbo Z nvme drive which claims a
sustained write speed of 6 GB/S. None of the existing file plugin see any
performance increase when writing to this device - I do not think that any
are actually write limited.  I have modified a raw binary file plugin that I
obtained from Keenan Lang that easily sustains the cameras write rate until
the device is full.

Problem is - Raw data really doesn't do anyone much good. I was thinking
that perhaps a quick solution to my problem might be to change this Raw File
plugin to look/act like a disk based fifo or circular buffer. This could
collect to the limit of the hardware at full speed, and if someone wanted
HDF output, they would just drain this queue at the speed that HDF files are
generated.  Or is there an easier/better solution? I.e. any way that file
plugins can use the new multi-thread model of AD 3.0?

I know the HDF files can be generates at very high speeds on Lustre file
systems, but this seems to be using parallel HDF5. Is this something that
Area Detector supports?


Replies:
Re: Area Detector and high performance NVME devices ulrik.pedersen
References:
Area Detector and high performance NVME devices Mark S. Engbretson
RE: Area Detector and high performance NVME devices Mark Rivers
RE: Area Detector and high performance NVME devices Mark S. Engbretson
RE: Area Detector and high performance NVME devices Mark Rivers

Navigate by Date:
Prev: RE: Area Detector and high performance NVME devices Mark Rivers
Next: PVs disconnected in CSS BOY despite access through caget Iain Marcuson
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  <2017
Navigate by Thread:
Prev: RE: Area Detector and high performance NVME devices Mark Rivers
Next: Re: Area Detector and high performance NVME devices ulrik.pedersen
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  <2017
ANJ, 28 Jun 2017 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·