Experimental Physics and
Hello Matt and Mark,
I've downloaded and installed the modified version of pyepics (with new use_numpy), and can confirm that the performance became a lot (8 times?) better. My program is now able to successfully receive (at most) 5 callbacks for subsequent frames made in 32 FPS mode, which is enough for my current needs.
However, out of 50 frames I still get callbacks for only 18, and it become worse as the amount of frames taken grows (so I'm unable to make pyepics videostream player :( ). If you have any further suggestions to boost this process (or guesses about the shortcomings I may be suffering from), I'll be glad to try them just to get rid of possible limitations in future. For example, as I understand, when I add a callback for a PV object with array data within, I become "subscribed" to the entire array. The problem may be that, as described below, the PV image data is always seen as large as (in my case) 1228800 elements array, while the actual region of interest size can be much smaller (all irrelevant elements of the array simply being set to 0). So, is it possible to create the PV object and the callback for it that will only monitor the specified part of itself (like, I assume, camonitor -# does)?
I've further tried to tweak the AreaDetector to take advantage of ROI to limit the amount of data to be processed (and to speed up the callbacks). I enabled NDROI plugin and set the lower right quarter of an input image as a region of interest. Then I assigned 'ROI1' as the array port to NDStdArrays. NDStdArrays plugin window correctly shows me array size of 640x480 (1/4th of 1280x960). I'm not surprised that AreaDetector itself continues to capture full image (thus no framerate increase), I can change that by setting Region start and Region size values in the Readout section of prosilica.adl window. The thing I don't clearly understand, though, is that using 'cainfo 13PS1:image1:ArrayData' shows me 'Element count: 1228800' no matter what I do, and the same remains the size of my pyepics PV object despite of array size being four times less. I've found that the amount of elements shown by cainfo depends on the line in st.cmd file:
Should I have NELEMENTS set to the correct value on startup to match the size of my ROI? If so, how does it connect with the fact that the size of ROI can be changed dynamically at any time, while cainfo will continue to show the same obsolete element count? What is the "true" size of '13PS1:image1:ArrayData' PV data that is requested through, for example, caget? I have found that caget tries to get large array even if I set image size to 1x1 in AreaDetector, and it correctly gets one pixel value after I set NELEMENTS to 1 in st.cmd file. If the NDStdArrays plugin only writes the required amount of data (and caget only requests the "true" amount of data) to the ArrayData PV, I can track the actual data size in my software. But if my various PV operations always request the full length of an image (as caget does), then I won't be able to take advantage of all the ROI stuff. Is it possible to change the size of data in PV "at the runtime"?
Thank you for all your assistance and invaluable help,
From: Matt Newville <firstname.lastname@example.org>
To: Anton Derbenev <email@example.com>
Cc: Mark Rivers <firstname.lastname@example.org>; "email@example.com" <firstname.lastname@example.org>
Sent: Wednesday, December 28, 2011 8:28 AM
Subject: Re: pyepics: Strategies for working with fast-changing large arrays
Thanks for all the detective work! OK, the conversion to numpy arrays
is definitely the bottleneck. It sort of surprises me, but the
evidence is pretty convincing.
I think a decent workaround is to change ca.py, _unpack function
(around line 860) to be
use_numpy = (HAS_NUMPY and as_numpy and ntype != dbr.STRING and
count > 1 and count < AUTOMONITOR_MAXLENGTH)
use_numpy = (count > 1 and HAS_NUMPY and as_numpy and ntype != dbr.STRING)
that is, to suppress conversion to numpy arrays for arrays larger than
ca.AUTOMONITOR_MAXLENGTH (default of 16384)
After reading your original message, I was confused because I know I
had tested fetching and displaying images from a 1.4Mb Prosilica
camera, and could definitely read at rates better than ~10Hz and even
display with wx (known to be slow for this) at better than ~5Hz.
But the last time I tested that was well over a year ago, and ...
it turns out that ca.py **used** to include the 'count < AUTOMONITOR_MAXLENGTH'
test for automatically using numpy in conversion of CA data to python
data, and I took this out with several other changes, about a year ago
(Jan 11, 2011 -- git commit 66317f8478). It should definitely go
I'm away from machines convenient for testing for the next couple
weeks, so I'm reluctant to push a commit to the master branch, but I'm
pretty sure this will work (I think your tests already confirmed
this). You could add this change by hand, or if you're interested,
you could checkout the 'pvget_improvements' branch from github , which
is the most recent development changes (for other issues with get()!),
and to which I've committed this change.
Longer term, it would be interesting to look at whether 'use_numpy'
should be used less aggressively or fully customizable per
PV/channelID. The testing a global setting with the array count
Anyway, thanks for the debugging work, sorry for the trouble, and
hope this helps,
On Tue, Dec 27, 2011 at 5:55 AM, Anton Derbenev <email@example.com> wrote:
> Hello Mark and Matt,
> thank you for your swift reply. I have considered your advices and questions
> and here what I've got:
> I am aware that in pyepics large waveforms are not automatically monitored
> by default, so I used auto_monitor=True to make that callbacks called.
> I see now that callbacks run in a separate thread. However, I want to use
> Python multiprocessing not because PV callback processing somehow interferes
> with main interpreter thread, but because my application has to do some
> other things (for example, GUI drawing, image processing) while receiving
> new frame data in the same time.
> Although I was pretty sure that the network is capable to bear the expected
> frame rate (because AreaDetector netCDF plugin saves every frame to disk
> without any problems), I used camonitor (camonitor -#2
> 13PS1:image1:ArrayData) as you suggested. It has successfully "captured" 100
> of 100 frames (printed 100 lines) with 10 FPS over 100Mbit network. I have
> also written this simple program that fails to repeat the success of
> camonitor through PV callbacks:
> import epics
> from epics import PV
> counter = 0
> def onArrayDataCallback(**kw):
> global counter
> print 'Callback #', counter + 1, ' called!'
> counter = counter + 1
> if __name__ == "__main__":
> array_data_pv = PV('13PS1:image1:ArrayData', callback =
> onArrayDataCallback, auto_monitor = True)
> global counter
> while counter < 10:
> epics.poll(evt=1.e-5, iot=0.1)
> I used AreaDetector to get series of images from my Prosilica camera in Free
> Run mode (~10 FPS, 1280x960x8bpp). The program above "captures" (i.e.
> executes callback for) 7 out of 10 images, the last output from it being
> "Callback # 7 called!". Out of 100 images, the callback function was only
> called 17 times!
> So, the problem is indeed that my Python callback is missing some events.
> Following the advice, I tried "profiling" the conversion of data from C to
> Python ( ca._unpack() ):
> from epics import ca
> import time
> chid = ca.create_channel('13PS1:image1:ArrayData')
> t = time.time()
> value = ca.get(chid, as_numpy = True)
> print 'Time:', time.time() - t
> I measured the time it takes to complete a single call to ca.get(...). It
> ranges from 0.75 to 0.82 sec, most of the time being taken by
> ca._unpack(...) - from 0.70 to 0.76 sec with as_numpy set to True. With
> as_numpy set to False, ca.get(...) takes 0.09 to 0.11 sec to complete, 0.04
> to 0.06 sec from this time being taken by ca._unpack(), i.e. almost 8 times
> So, what can I do to make my Python callback not to miss any events? Maybe
> if the PV is created with auto_monitor = True as in my first test program
> above, the waveform data is then unpacked with as_numpy = True by default?
> If so, and if converting to numpy is an issue, how can I create the
> auto-monitored PV so that it doesn't attempt to convert the value to numpy
> array when its callback is invoked?
> Also when I said that I were "omitting EPICS at all", I meant that my
> current workaround doesn't use the PV or ca. Application runs on the same
> machine as the IOC and uses netCDF files that were written to the disc by
> the AreaDetector.
> I don't really know where I can check the datatype (FTVL) and dimensions of
> the EPICS waveform record I am using, but "the numerical CA type indicating
> the data type" for my data is "4" (that's what my Python program prints),
> whatever it means. Also I tried to set the region size in AreaDetector to
> 16x16 elements, 8bpp, so that image size was 256 bytes. In spite of this,
> the size of data (in elements) inside the PV object in my Python program
> remained 1228800 (1280x960). So I guess that you are right, I get the entire
> array on every callback regardless of AreaDetector ROI configuration. It
> doesn't seem strange for me itself (though I don't know how to obtain only
> the part of the array in pyepics), but I'd prefer to obtain all the data
> (full 1280x960x12bpp) anyways.
> I tried the approach that you described, i.e. subscribing to a "fast" PV,
> like frame counter, so that no callbacks are missed. There are some
> difficulties for me with pyepics CA contexts, though (like not being able to
> get the value of one PV inside the callback function of another or getting
> corrupted image data from PV.get(...) function).
> At 1280x960x8bpp resolution over 10 Mbit network, I get approx. 10 FPS
> either in AreaDetector free run mode or in Prosilica application. That is
> the maximum with current network card.
> I am running Debian.
> I firmly believe that the camera and AreaDetector are capable of getting
> image data fast enough (given 10Mbit), so I'd expect that ImageJ and the IDL
> viewer show the same that Prosilica applications shows. The question is, why
> callbacks fail to invoke.
> As I said above, I've tried running camonitor and it did just fine not
> missing a single callback. Not that my application do the same.
> Best regards,
> Derbenev Anton.
> ----- Original Message -----
> From: Mark Rivers <firstname.lastname@example.org>
> To: Matt Newville <email@example.com>; Anton Derbenev
> Cc: firstname.lastname@example.org
> Sent: Saturday, December 24, 2011 5:27 AM
> Subject: RE: pyepics: Strategies for working with fast-changing large arrays
> Hi Anton,
> I have not used the Python EPICS interface with areaDetector, but here are
> some additional questions and suggestions:
> - What is the datatype (FTVL) and dimensions of the EPICS waveform record
> you are using? Is it sized to be just the required size for the detector?
> If it is much larger then your callback could be wasting bandwidth and CPU
> cycles because you are probably subscribing for callbacks for the entire
> array, not just the required size.
> - In the ImageJ and IDL viewers I don't put monitors on the array, but
> rather on the frame counter longin record. When that increments I do a
> caget of the waveform record, but only requesting the number of bytes
> required for the current image, which can dynamically change as the ROI,
> datatype and color mode are changed.
> - If you use the ImageJ plugin that comes with areaDetector what frame rate
> do you get? It can display at least 30 frames/sec at the size camera you
> have. I usually use 8-bits, not 12 (which is really 16), but it should
> still be able to do about the same rate.
> - Are you running on Linux or Windows?
> - You can also try the IDL viewer, which can be run for free with the IDL
> Virtual Machine. It can also do rates about the same as ImageJ.
> - As Matt suggested try camonitor, but just get a few array values, e.g.
> camonitor -#2 13PS1:image1:ArrayData
> See how quickly those callbacks arrive.
> From: email@example.com on behalf of Matt Newville
> Sent: Fri 12/23/2011 8:44 AM
> To: Anton Derbenev
> Cc: firstname.lastname@example.org
> Subject: Re: pyepics: Strategies for working with fast-changing large arrays
> Hi Anton,
> On Fri, Dec 23, 2011 at 3:38 AM, Anton Derbenev <email@example.com>
>> Hello all,
>> recently I have tried to make a Python application using Epics Channel
>> Access for Python.
>> I am using an EPICS areaDetector to operate a Prosilica digital camera,
>> from which I need to rapidly obtain several images.
>> Getting a single PV waveform data snapshot itself is not an issue and
>> works just fine the way it is described in the pyepics documentation.
>> Although a call to PV.get(...) for 1280x960x12bit data array blocks the
>> program execution for too long (relatively to camera FPS), I've managed to
>> get rid of any delay from my software by utilizing a multiprocessing Python
>> interface to avoid Global Interpreter Lock.
>> Thus, my application is capable of running several data handling functions
>> simultaneously without blocking.
>> That's where the bad part starts: when the PV that holds the image data is
>> set to be automatically monitored (auto_monitor = True), the callback
>> invocation happens too slow.
>> For example, when the camera is triggered and makes, like, 5 shots in one
>> second, the callback for corresponding PV only runs three times!
>> I tried both "high-level" (PV object) and "low-level" (channel access
>> contexts) approaches trying to make the callback calling to be faster, but
> Sorry for the trouble. There have been several conversations over the
> past few months about getting waveform records with pyepics. Some
> changes were made fairly recently (to version 3.1.4), and there are
> some changes in a branch on github (though I think it won't help your
> case). We can make more changes if they're needed, but at this point,
> I don't know what those might be. It would be nice to understand what
> you're doing and why it's not working.
> For reference, a waveform PV with count larger than
> ca.AUTOMONITOR_MAXLENGTH (default
> 16384) is *not* automatically monitored, unless you use
> PV(name, ..., auto_monitor=True)
> I read your message to mean that you did use 'auto_monitor=True'. If
> that's the case, the data will be sent to the PVs internal callback by
> the CA library. A PV.get() should simply return the value received
> from the most recent monitor callback.
> If auto_monitor is not set for a large waveform, then a PV.get() (or
> ca.get() with the Channel ID) will call the C ca_array_get_callback()
> function, and then wait for the callback to complete.
> In either case, the callback runs in a separate thread, asynchronous
> (assuming pre-emptive callbacks) from the main Python thread and
> should not be slowed down by anything to do with Python (for example,
> the GIL). The data sent from the CA callback or
> ca_array_get_callback() does have to be converted from the C structure
> to a Python list or numpy array, but it's hard to believe that cannot
> keep up with 5 frames per second for a 1Mb image. A few diagnostic
> a) verify with command-line camonitor that you're getting the
> expected frame rate over your network.
> b) write a test callback function for the PV object that simply
> records or prints when it gets a new value for the waveform.
> Those should help isolate the problem. If the problem is that your
> Python callback is missing some events, you might try profiling the
> conversion of data from C to Python (ca._unpack(), especially the
> conversion to numpy arrays). I'd be surprised that was the culprit,
> but it's easier to believe than most other options.
> Again, I doubt that Python's GIL has anything to do with this, as most
> of the processing happens in the CA library. I am also not sure that
> multiprocessing would actually help. By default, pyepics tries to
> ensure that a single CA thread context is used. Depending on how you
> use it, multiprocessing might increase the network traffic.
>> I assume that every time when the automatically monitored PV changes, its
>> status is requested ( like in PV.get(...) ) and then passed as argument to
>> callback function.
> Actually, a monitored PV is registered with the CA library, and a
> callback function is run when the PV changes. This happens from the C
> library, in a thread separate from and asynchronous to the main python
>> So if a CA (channel access) does not allow rapid subsequent queries to a
>> single PV, hence the delay that limits the amount of callbacks that can be
>> called per second, this amount being less then my camera's FPS.
>> If this is so, then setting the auto_monitor to False and simply polling
>> PV with desired frequency will not work either (also this approach causes
>> amounts of unnecessary network traffic).
> Using 'auto_monitor=False' will result in the least amount of network
> traffic. You would need to do an
> explicit PV.get() (which calls ca_array_get_callback()) to get a new
> value for a PV. If you don't ask for data fast enough, you will miss
> some data.
>> Thanks to the mighty areaDetector software, I have found the workaround
>> (omitting EPICS at all), but it involves the file system of the hardware
>> the IOC runs, thus being not completely satisfactory.
> Perhaps I'm not understanding -- how can you be using areaDetector and
> omitting EPICS at all? Anyway, if the data can be written to disk at
> a sufficient speed, any program ought to be able to keep up... Are
> you running the pyepics program on the same machine as the IOC or a
> different machine?
>> Hereby I require aid: what are the strategies for working with
>> large data arrays using pyepics? How can I make my camera frame data PV
>> object to invoke callbacks fast enough or, if it is not possible, what are
>> other ways to save and later access the value of such PVs using EPICS?
>> Best regards,
>> Derbenev Anton.
> Hope that helps.
--Matt Newville <newville at cars.uchicago.edu> 630-252-0431
|ANJ, 18 Nov 2013||
· EPICSv4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·