Experimental Physics and
Industrial Control System

Matt Newville <[email protected]> · Sun, 4 Oct 2009 11:38:52 -0500

 Hi Matthieu,

On Fri, Oct 2, 2009 at 9:30 PM, Matthieu Bec <[email protected]> wrote:
>
> Hi Matt,
>
>> Of course.  But Python does not have separate native datatypes for
>> int, short, long and does not distinguish unsigned and signed integer
>> types (there are extensions that can make this distinction, mostly
>> used to pack data for other C libraries).   So, the Python programmer
>> should never be forced to make the distinction, or even have to know
>> that it exists.
>>>>
>>>> The DBR_ type for a channel is an implementation detail that is
>>>> important in C, but not in Python.
>
> I think it depends on what you use it for. Consider 'numpy', that has more
> data types. It's not python, but numpy is quite an essential add-on: I use
> it so much I would personally not mind see it (at C-API level) used in
> python-ca.

I completely agree that one wants to get CA array data into numpy
arrays quickly and efficiently.

Perhaps I was unclear earlier: I think a Python interface to CA should
fetch the native CA type (or perhaps the CTRL or TIME variant on
special request) and then convert that into Python objects in the
interface layer.

For an array of doubles (DBR_FLOAT,DBR_DOUBLE) or ints
(DBR_SHORT,DBR_LONG), automatically converting that to a numpy array
of the appropriate "dtype"  makes perfect sense. If one does a
ca_put() with a numpy array on a PV that is natively DBR_FLOAT, it
would be nice if the interface layer made the conversion.

Similarly, for waveform records of characters, turning that into a
string makes the most sense to me. I can see that other conversions of
character waveforms might be desirable, but not supporting any
conversions for the sake of "being compete and general" seems like a
bad choice to me.

The point I was trying to make was that a "complete C API" is a bit
more general than needed.  For example, request_type != native_type is
not needed (again, CTRL and TIME would variants are needed).  I'm not
opposed to allowing a Python programmer to say "I know this PV is a
DBR_FLOAT, but I want to get it as a DBR_SHORT", I just see it as
pointless.  DBR_SHORT is a detail that is important for C (and a
Python interface has to deal with this), but it is not important for
Python.  Again, you *might* want a numpy array, and this should be of
the correct type: the interface should known that an Epics array of
DBR_SHORT corresponds to a numpy array with dtype=uint16.  If a
"complete API" means that the Python programmer *has* to deal with a
DBR_*** type, then it's bad Python.

>> In addition, mixing threads well between C and Python is well known to
>> be hard and error-prone.    A "complete" API would probably need to
>> allow "ca_create_context(ca_enable_preemptive_context)". I'm not sure
>> that even make sense with a language with its own VM.  How is this
>> *supposed* work in such a case?
>
> so, I wrote and maintain my own extension (missing from your survey :) that
> actually does that. Here are the use cases that motivated me:

I guess it's just so easy to roll owns one interface that there is
very little incentive to use someone else's code even if it is
available and documented.  Coming to a common solution would be nice.

> - python used as a shell
> I found ca_enable_preemptive_context + python GIL work well.
> early implementation (R3.12? when ca was not thread safe) used hooks to
> libreadline and handle the ca background polling.

As I understand it, "preemptive callback" was introduced in 3.14 and
means that one does not need to poll; there is a C thread effectively
polling for CA events for you in the background. The issue (for
Python) is whether Python callback functions can be sensibly run from
the background CA polling thread without coordinating with the Python
main thread.  Python supports threads, but only allows one thread at a
time to have access Python objects in the global context of the
process.  If a background CA polling thread cannot acquire and release
the Python GIL, I don't understand how it could do anything useful (to
Python).   Perhaps I am misunderstanding.

For interactive shell work, I rarely use callbacks at all.  Do you use
callbacks here? I do like your idea of adding a polling hook into the
readline library so that one did not need to explicitly poll from the
shell, but that's the same as enabling preemptive callbacks.  Again,
maybe I am misunderstanding something here. Please correct me if I am
wrong.

> - python UI with your python-toolkit
> mixing X and threads seems extremely difficult, the simplicity one might be
> seeking with python script is lost. Here instead, I have a (gtk) wrapper
> that starts ca_disable_preemptive_context and handles ca polling with a
> glib.timeout_add(...)

My experience (and I believe the experience of everyone else using
Python and CA) is that one needs needs to be very careful of using
native CA threads and GUI-level threads, as mixing CA and GUI threads
through a Python main thread can easily crash.

Even with preemptive_callback disabled, and polling to run Python
callback functions (for PVs which have one defined), one has to be
careful as the Python callback is run entirely within that
ca_pend_event().  That means no other CA calls can be made, and that
calls into other threads (say, to update a widget) are disaster-prone.
One can access Python objects with confidence, but ca_pend_event()
often needs to be run quickly (so don't fetch data from a URL or
process an image!) The strategy I take with wxPython is to have a
python callback for each PV that simply notes that the PV has changed
(and caches the value), and then set up a Timer loop() for the GUI
that effectively does
    ca_pend_event()
    react_to_changes()

What I *really* do is sub-class widgets (PVStaticText, PVTextCtrl,
etc) that includes has the simple callback, knoww which GUI field to
update, and sets up (or reuses) a Timer loop as above.  That way, PV
values in widgets get updated automatically and only on real changes.
I believe that others using CA and widget toolkits do approximately
the same thing, though I'd be happy to hear of a better way to do it.

>> For what it's worth, My own extension and cothreads do not enable
>> preemptive callbacks, and neither has "native" threads -- my own
>> simply does not have them at all.
>
> Does cothreads let you handle those 3 use cases somewhat transparently? One
> big argument for scripting is that it makes things more straightforward:
> import the module, do the work.

I believe that all Python interfaces will work with these three use
cases. I won't speak for cothreads (I'm still looking at this code: I
very much like its use of ctypes) only for my own interface: EpicsCA.

For interactive shell: Yes, this works.  One needs to occasionally
poll().  I like yor idea of hiding a poll() in a readline hook.  From
an interactive shell, I rarely use callbacks and mostly use
higher-level functions caget() and caput() functions which include
polling anyway.

For GUI code: Yes, this works.

For scripts: Yes, this works.  I run many long running (months)
scripts that run a  "poll-and react to events" loop.

Cheers,

--Matt Newville

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System