Experimental Physics and
Industrial Control System

Matt Newville <[email protected]> · Tue, 27 Dec 2011 19:28:58 -0600

Anton,

Thanks for all the detective work!  OK, the conversion to numpy arrays
is definitely the bottleneck.  It sort of surprises me, but the
evidence is pretty convincing.

I think a decent workaround is to change ca.py, _unpack function
(around line 860) to be
    use_numpy = (HAS_NUMPY and as_numpy and ntype != dbr.STRING and
                 count > 1 and count < AUTOMONITOR_MAXLENGTH)

instead of
    use_numpy = (count > 1 and HAS_NUMPY and as_numpy and ntype != dbr.STRING)

that is, to suppress conversion to numpy arrays for arrays larger than
ca.AUTOMONITOR_MAXLENGTH (default of 16384)

After reading your original message,  I was confused because I know I
had tested fetching and displaying images from a 1.4Mb Prosilica
camera, and could definitely read at rates better than ~10Hz and even
display with wx (known to be slow for this) at better than ~5Hz.
But the last time I tested that was well over a year ago, and ...

it turns out that ca.py  **used** to include the 'count < AUTOMONITOR_MAXLENGTH'
test for automatically using numpy in conversion of CA data to python
data, and I took this out with several other changes, about a year ago
(Jan 11, 2011  -- git commit 66317f8478).   It should definitely go
back in.

I'm away from machines convenient for testing for the next couple
weeks, so I'm reluctant to push a commit to the master branch, but I'm
pretty sure this will work (I think your tests already confirmed
this).   You could add this change by hand, or if you're interested,
you could checkout the 'pvget_improvements' branch from github , which
is the most recent development changes (for other issues with get()!),
and to which I've committed this change.

Longer term, it would be interesting to look at whether 'use_numpy'
should be used less aggressively or fully customizable per
PV/channelID.   The testing a global setting with the array count
seems kludgy.

Anyway,  thanks for the debugging work, sorry for the trouble, and
hope this helps,

--Matt

On Tue, Dec 27, 2011 at 5:55 AM, Anton Derbenev <[email protected]> wrote:
> Hello Mark and Matt,
>
> thank you for your swift reply. I have considered your advices and questions
> and here what I've got:
>
> Matt,
>
> I am aware that in pyepics large waveforms are not automatically monitored
> by default, so I used auto_monitor=True to make that callbacks called.
>
> I see now that callbacks run in a separate thread. However, I want to use
> Python multiprocessing not because PV callback processing somehow interferes
> with main interpreter thread, but because my application has to do some
> other things (for example, GUI drawing, image processing) while receiving
> new frame data in the same time.
>
> Although I was pretty sure that the network is capable to bear the expected
> frame rate (because AreaDetector netCDF plugin saves every frame to disk
> without any problems), I used camonitor (camonitor -#2
> 13PS1:image1:ArrayData) as you suggested. It has successfully "captured" 100
> of 100 frames (printed 100 lines) with 10 FPS over 100Mbit network. I have
> also written this simple program that fails to repeat the success of
> camonitor through PV callbacks:
>
> import epics
> from epics import PV
>
> counter = 0
>
> def onArrayDataCallback(**kw):
>     global counter
>     print 'Callback #', counter + 1, ' called!'
>     counter = counter + 1
>
> if __name__ == "__main__":
>     array_data_pv = PV('13PS1:image1:ArrayData', callback =
> onArrayDataCallback, auto_monitor = True)
>     global counter
>     while counter < 10:
>         epics.poll(evt=1.e-5, iot=0.1)
>
> I used AreaDetector to get series of images from my Prosilica camera in Free
> Run mode (~10 FPS, 1280x960x8bpp). The program above "captures" (i.e.
> executes callback for) 7 out of 10 images, the last output from it being
> "Callback # 7 called!". Out of 100 images, the callback function was only
> called 17 times!
>
> So, the problem is indeed that my Python callback is missing some events.
> Following the advice, I tried "profiling" the conversion of data from C to
> Python ( ca._unpack() ):
>
> from epics import ca
> import time
>
> chid = ca.create_channel('13PS1:image1:ArrayData')
> t = time.time()
> value = ca.get(chid, as_numpy = True)
> print 'Time:', time.time() - t
>
> I measured the time it takes to complete a single call to ca.get(...). It
> ranges from 0.75 to 0.82 sec, most of the time being taken by
> ca._unpack(...) - from 0.70 to 0.76 sec with as_numpy set to True. With
> as_numpy set to False, ca.get(...) takes 0.09 to 0.11 sec to complete, 0.04
> to 0.06 sec from this time being taken by ca._unpack(), i.e. almost 8 times
> faster.
>
> So, what can I do to make my Python callback not to miss any events? Maybe
> if the PV is created with auto_monitor = True as in my first test program
> above, the waveform data is then unpacked with as_numpy = True by default?
> If so, and if converting to numpy is an issue, how can I create the
> auto-monitored PV so that it doesn't attempt to convert the value to numpy
> array when its callback is invoked?
>
> Also when I said that I were "omitting EPICS at all", I meant that my
> current workaround doesn't use the PV or ca. Application runs on the same
> machine as the IOC and uses netCDF files that were written to the disc by
> the AreaDetector.
>
> Mark,
>
> I don't really know where I can check the datatype (FTVL) and dimensions of
> the EPICS waveform record I am using, but "the numerical CA type indicating
> the data type" for my data is "4" (that's what my Python program prints),
> whatever it means. Also I tried to set the region size in AreaDetector to
> 16x16 elements, 8bpp, so that image size was 256 bytes. In spite of this,
> the size of data (in elements) inside the PV object in my Python program
> remained 1228800 (1280x960). So I guess that you are right, I get the entire
> array on every callback regardless of AreaDetector ROI configuration. It
> doesn't seem strange for me itself (though I don't know how to obtain only
> the part of the array in pyepics), but I'd prefer to obtain all the data
> (full 1280x960x12bpp) anyways.
>
> I tried the approach that you described, i.e. subscribing to a "fast" PV,
> like frame counter, so that no callbacks are missed. There are some
> difficulties for me with pyepics CA contexts, though (like not being able to
> get the value of one PV inside the callback function of another or getting
> corrupted image data from PV.get(...) function).
>
> At 1280x960x8bpp resolution over 10 Mbit network, I get approx. 10 FPS
> either in AreaDetector free run mode or in Prosilica application. That is
> the maximum with current network card.
>
> I am running Debian.
>
> I firmly believe that the camera and AreaDetector are capable of getting
> image data fast enough (given 10Mbit), so I'd expect that ImageJ and the IDL
> viewer show the same that Prosilica applications shows. The question is, why
> callbacks fail to invoke.
>
> As I said above, I've tried running camonitor and it did just fine not
> missing a single callback. Not that my application do the same.
>
> Best regards,
> Derbenev Anton.
>
> ----- Original Message -----
> From: Mark Rivers <[email protected]>
> To: Matt Newville <[email protected]>; Anton Derbenev
> <[email protected]>
> Cc: [email protected]
> Sent: Saturday, December 24, 2011 5:27 AM
> Subject: RE: pyepics: Strategies for working with fast-changing large arrays
>
> Hi Anton,
>
> I have not used the Python EPICS interface with areaDetector, but here are
> some additional questions and suggestions:
>
> - What is the datatype (FTVL) and dimensions of the EPICS waveform record
> you are using?  Is it sized to be just the required size for the detector?
> If it is much larger then your callback could be wasting bandwidth and CPU
> cycles because you are probably subscribing for callbacks for the entire
> array, not just the required size.
>
> - In the ImageJ and IDL viewers I don't put monitors on the array, but
> rather on the frame counter longin record.  When that increments I do a
> caget of the waveform record, but only requesting the number of bytes
> required for the current image, which can dynamically change as the ROI,
> datatype and color mode are changed.
>
> - If you use the ImageJ plugin that comes with areaDetector what frame rate
> do you get?  It can display at least 30 frames/sec at the size camera you
> have.  I usually use 8-bits, not 12 (which is really 16), but it should
> still be able to do about the same rate.
>
> - Are you running on Linux or Windows?
>
> - You can also try the IDL viewer, which can be run for free with the IDL
> Virtual Machine.  It can also do rates about the same as ImageJ.
>
> - As Matt suggested try camonitor, but just get a few array values, e.g.
>
> camonitor -#2 13PS1:image1:ArrayData
>
> See how quickly those callbacks arrive.
>
> Cheers,
> Mark
>
>
>
>
> ________________________________
>
> From: [email protected] on behalf of Matt Newville
> Sent: Fri 12/23/2011 8:44 AM
> To: Anton Derbenev
> Cc: [email protected]
> Subject: Re: pyepics: Strategies for working with fast-changing large arrays
>
>
>
> Hi Anton,
>
> On Fri, Dec 23, 2011 at 3:38 AM, Anton Derbenev <[email protected]>
> wrote:
>> Hello all,
>>
>>
>> recently I have tried to make a Python application using Epics Channel
>> Access for Python.
>>
>> I am using an EPICS areaDetector to operate a Prosilica digital camera,
>> from which I need to rapidly obtain several images.
>>
>> Getting a single PV waveform data snapshot itself is not an issue and
>> works just fine the way it is described in the pyepics documentation.
>>
>> Although a call to PV.get(...) for 1280x960x12bit data array blocks the
>> program execution for too long (relatively to camera FPS), I've managed to
>> get rid of any delay from my software by utilizing a multiprocessing Python
>> interface to avoid Global Interpreter Lock.
>>
>> Thus, my application is capable of running several data handling functions
>> simultaneously without blocking.
>>
>> That's where the bad part starts: when the PV that holds the image data is
>> set to be automatically monitored (auto_monitor = True), the callback
>> invocation happens too slow.
>>
>> For example, when the camera is triggered and makes, like, 5 shots in one
>> second, the callback for corresponding PV only runs three times!
>> :(
>>
>> I tried both "high-level" (PV object) and "low-level" (channel access
>> contexts) approaches trying to make the callback calling to be faster, but
>> no
>> luck.
>
> Sorry for the trouble.  There have been several conversations over the
> past few months about getting waveform records with pyepics.  Some
> changes were made fairly recently (to version 3.1.4), and there are
> some changes in a branch on github (though I think it won't help your
> case).  We can make more changes if they're needed, but at this point,
> I don't know what those might be.  It would be nice to understand what
> you're doing and why it's not working.
>
> For reference, a waveform PV with count larger than
> ca.AUTOMONITOR_MAXLENGTH (default
> 16384) is *not* automatically monitored, unless you use
>   PV(name, ..., auto_monitor=True)
>
> I read your message to mean that you did use 'auto_monitor=True'.  If
> that's the case, the data will be sent to the PVs internal callback by
> the CA library.  A PV.get() should simply return the value received
> from the most recent monitor callback.
>
> If auto_monitor is not set for a large waveform, then a PV.get() (or
> ca.get() with the Channel ID) will call the C ca_array_get_callback()
> function, and then wait for the callback to complete.
>
> In either case, the callback runs in a separate thread, asynchronous
> (assuming pre-emptive callbacks) from the main Python thread and
> should not be slowed down by anything to do with Python (for example,
> the GIL).  The data sent from the CA callback or
> ca_array_get_callback() does have to be converted from the C structure
> to a Python list or numpy array, but it's hard to believe that cannot
> keep up with 5 frames per second for a 1Mb image.  A few diagnostic
> tests:
>
>   a) verify with command-line camonitor that you're getting the
> expected frame rate over your network.
>   b) write a test callback function for the PV object that simply
> records or prints when it gets a new value for the waveform.
>
> Those should help isolate the problem. If the problem is that your
> Python callback is missing some events,  you might try profiling the
> conversion of data from C to Python (ca._unpack(), especially the
> conversion to numpy arrays).  I'd be surprised that was the culprit,
> but it's easier to believe than most other options.
>
> Again, I doubt that Python's GIL has anything to do with this, as most
> of the processing happens in the CA library.  I am also not sure that
> multiprocessing would actually help. By default, pyepics tries to
> ensure that a single CA thread context is used.  Depending on how you
> use it, multiprocessing might increase the network traffic.
>
>> I assume that every time when the automatically monitored PV changes, its
>> status is requested ( like in PV.get(...) ) and then passed as argument to
>> the
>> callback function.
>
> Actually, a monitored PV is registered with the CA library, and a
> callback function is run when the PV changes.  This happens from the C
> library, in a thread separate from and asynchronous to the main python
> thread.
>
>> So if a CA (channel access) does not allow rapid subsequent queries to a
>> single PV, hence the delay that limits the amount of callbacks that can be
>> called per second, this amount being less then my camera's FPS.
>>
>> If this is so, then setting the auto_monitor to False and simply polling
>> the
>> PV with desired frequency will not work either (also this approach causes
>> large
>> amounts of unnecessary network traffic).
>
> Using 'auto_monitor=False' will result in the least amount of network
> traffic.  You would need to do an
> explicit PV.get() (which calls ca_array_get_callback()) to get a new
> value for a PV.  If you don't ask for data fast enough, you will miss
> some data.
>
>> Thanks to the mighty areaDetector software, I have found the workaround
>> (omitting EPICS at all), but it involves the file system of the hardware
>> where
>> the IOC runs, thus being not completely satisfactory.
>
> Perhaps I'm not understanding -- how can you be using areaDetector and
> omitting EPICS at all?  Anyway, if the data can be written to disk at
> a sufficient speed, any program ought to be able to keep up...  Are
> you running the pyepics program on the same machine as the IOC or a
> different machine?
>
>> Hereby I require aid: what are the strategies for working with
>> fast-changing
>> large data arrays using pyepics?  How can I make my camera frame data PV
>> object to invoke callbacks fast enough or, if it is not possible, what are
>> other ways to save and later access the value of such PVs using EPICS?
>>
>> Best regards,
>> Derbenev Anton.
>
> Hope that helps.
>
> --Matt
>
>
>

-- 
--Matt Newville <newville at cars.uchicago.edu> 630-252-0431

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System