Experimental Physics and
Industrial Control System

"Steven M. Hartman" <[email protected]> · Wed, 29 Sep 2010 16:55:31 -0400

[email protected] wrote:
I like the idea of having a RDB to do channel archiving.
But compared to files it obviously increases the complexity.

Yes, the complexity is an issue.
For the SNS, the complexity of handling the data was the issue
which led to the RDB based archiver. With the previous file based
archiver and a 2GB file size limit, the difficulty was in dealing
with lots of sub-archives, moving data from the archive engine
servers to a data server for client retrieval, lots of archive
engines each with their local configuration, long term disk
storage issues, etc. It was also difficult to understand what was
actually going into the archiver at what rate and producing how
much data. Moreover, options for decimating or selectively purging
old data were pretty much not possible.
The RDB back-end simplifies much of this. Let the RDB handle the
data. Additionally, keep all of the archive engine configurations
in a central database. Now it becomes possible to use standard
database techniques to look at the data and the configuration:
which PVs have a sample in this period of time? how many samples
has a particular archive engine inserted in the past 2 hours?
which archive engine is archiving this PV (while ensuring that
only _one_ engine actually is archiving it)? Simple web tools can
be used to review or update the archive engine configurations in
the RDB. SQL can be used to retrieve the data. (Decimating or
selectively purging old data and long term storage are still
issues, though. The primary issue is more of policy: deciding what
and when. The RDB makes it technically possible, but there is
still uncertainty with actual performance of purging)
But the trade-off with the ease in data handling is the increase
complexity in the initial setup. You need an RDB and maybe a DBA.
You find issues with 'extending table space' and 'redo log files'
and other fun stuff.
I went through the doc and it says that Oracle can handle 8000 pv updates per second.
Were any benchmark done with a mysql backend?

Yes, but I don't have the numbers. Short answer: mysql was faster,
but less extensible.
Ultimately the population of the database (creation of tables) should be 'automated' and starting from a
'dbl <recordType> > ... '.

It's certainly possible, but in the case of SNS, we don't
necessarily want all ~500,000 records to be archived.
Tying it in with irmis tables should also be possible if you are
interested in trying that.
But the doc mentions we should be using eclipse and this and that.
For this simple operation, that seems a little complicated and overkill...

The archive engine is a 'headless' css application so that implies
eclipse.
--
Steven Hartman
[email protected] || 865-466-6473

Experimental Physics and Industrial Control System

Experimental Physics and
Industrial Control System