Experimental Physics and
Industrial Control System

Matt Newville <[email protected]> · Mon, 25 Jun 2012 11:50:53 -0500

Hi Miroslav,

On Sat, Jun 23, 2012 at 4:37 PM, Mihaylov, Miroslav N. <[email protected]> wrote:
> Hello.
> I would like to share my experience on storing, processing and visualizing
> time series data.
> I am a physics graduate student from UIC (University of IL at Chicago).
>
> The application.
>
> http://131.193.191.37/~mnm/temperatures/menu.php

That looks nice.  I like the interactive zooming,  but I'm not exactly
sure I understand what I'm looking at.   For me, the plots seems to
dance around quite a bit, sometimes showing a long history of a couple
temperatures, sometimes showing nearly nothing...  Maybe I'm not using
it correctly?

> Let me explain what that is.
> Our research  group has a beam-time this week at ChemMatCARS Sector-15 at
> APS.
> We wanted to look at the channels A and B of our own lakeshore 340 and
> thermal probe connected to Keithley 2000.  Both connected to our laptop
> via GPIB – USB cable.
> Using python with pyVisa we read those 3 variables and   write them  to
> the MySQL database of the desktop that is located  at  UIC- that is the
> link above. The connection to the UIC desktop is via the guest wirelesses
> and SSH tunnel.
> This web application is a stripped down version of something I am
> currently developing for our in-house experiment at UIC that is being
> controlled with  EPICS.

If you're using PyVisa to read the meter over GPIB/USB, that suggests
these temperatures are not Epics Variables.   Is that correct?   You
can run a Keithley meter with Epics and read the temperatures with a
Channel Access client at fairly high speed.  Is this something you are
doing, or have in mind to do?

> In general there is a need for visualization of real time data in the
> scientific community.  However setting up and maintaining the
> infrastructure is not a trivial matter. With this kind of centralized
> system this problem could be significantly reduced.
> For example for an EPICS environment such as the beamlines at APS the task
> of visualizing small number of PV s could be reduced down to running a
> simple python script on the client machine given that pyEpics installed on
> the client machine.   The database server has to be setup only once at one
> location.
> My rough estimate is that a single modern  desktop system can serve tens
> of clients simultaneously each individually recording at a rate of 10Hz.

Yes, pushing changes in PV values into a database server can go very
fast.   And, yes, using a database backend is a good way to centralize
data collection.....

> The main requirement is fast hard drive array.
>
> I have been thinking for a while starting a project on github to make this
> web based application more general and would like to see if there is
> anybody else doing something similar and would like to collaborate on
> that.

Can you give a few more details of what you have in mind?    You might look at
    https://github.com/newville/epicsarchiver (for code) and
http://cars9.uchicago.edu/cgi-bin/pvarch/ (for example installation)

of a 'python/epics/mysql' data archiver with a web interface.  It
could definitely use some attention especially for faster, more
interactive web and graphical displays of data, etc.    I'm not sure
it's exactly what you have in mind, but it might be worth looking at.

Cheers,

--Matt Newville <newville at cars.uchicago.edu> 630-252-0431

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System