EPICS Home

Experimental Physics and Industrial Control System


 
1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  <20172018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  <20172018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: Questions about Archiver Appliance deployment
From: Michael Davidsaver <[email protected]>
To: "Wang, Lin" <[email protected]>
Cc: [email protected]
Date: Mon, 10 Apr 2017 11:32:19 -0400
On 04/10/2017 10:33 AM, Wang, Lin wrote:
> Dear all,
> 
> We plan to migrate from Channel Archiver to Archiver Appliance for archiving in CSNS (China Spallation Neutron Source) accelerator control system.
> 
> Our platform for Archiver Appliance deployment is:
> Debian Jessie 8.7, HP blade server with 512G memory and 20-total-core Xeon E5-2650 processor.
> 
> After a couple of weeks' evaluation (deployed using single_machine_install.sh), I have some questions as follows.
> 
> 1. For the 512G memory, in order to have a good performance, how much memory shall I allocate to RAMDisk (for STS) and how much to JVM heap (i.e. -Xms & -Xmx options) of each of the 4 components?

I not sure I can answer this directly as this will depend on the rate of
updates.  My advise would be to learn about the 'jvisualvm' utility and
using it to monitor cpu and memory usage of the 4 components.  Also
watch the log files.  If things go very wrong you'll see out of memory
exceptions being logged.

> 2. We have two blade servers dedicated for Archiver Appliance deployment and will only archive less than 100,000 channels, so we prefer redundancy (failover) instead of cluster. Is there any information about Archiver Appliance redundancy deployment?
> 
> 3. It seems that Archiver Appliance allows users to achieve configuration and operation via web UI and script (HTTP/Json), but it is not what we want in our control network. So how can I enforce restrictions to the web UI or web service so that only admin can manage?
> 
> 4. We need to convert Channel Archiver files to Archiver Appliance PB files for a complete transition, so which tool will be recommended to do this? Is the ca2aa [1] still available or already obsolete? I saw there is no README or wiki for that repo any more.

I haven't used it since I left BNL in 2015.  I don't think the PB file
format has changed in this time, so it should still work.  How much data
do you plan to convert?

There are two converters available.  One included with carchivetools
itself.  This goes through the channelarchiver xmlrpc interface and is
quite slow.  The other is ca2aa which you found, which directly converts
channelarchiver files and is much faster.

I see that there is no mention in the documentation.  For the first, at
this point the best I can provide is in the 'test-pb-export.sh' file in
the carchivetools repository.

For ca2aa it's simple.  Just point it to a channelarchiver index file
and give a directory where the .pb files will be written.

> ./exportall.py /some/ca/index/file /pb/output/dir/

You may also want to investigate the '--seps' and '--pvlist' arguments.

> arget -s "2014-11-10 21:30:00" -e "2014-11-10 21:35:00" --appliance-name appliance1 -E pbraw --export-granularity 1month --export-out-dir /home/jbobnar/pbraw --search -W manyChannel_*

> 5. Is there any other tool or script that is useful for performance or reliability test for Archiver Appliance?

For performance testing, raw retrieval speed can simply be measured with
an http client like 'curl'.  I also used my carchivetools scripts for
cross-comparisons between channelarchiver and archiver appliance (AA is
much faster).  As a hint, if you use 'arget' with very verbose debugging
you will see the URLs being fetched.  You can then use this with 'curl'.

For reliability, I used my script to retrieve and store particular PVs
and time range several times and compare the results.  Doing this before
and after a time range is migrated gives some confidence that the data
migration process wasn't corrupting anything.

> Any suggestion is appreciated.
> 
> 
> Thanks,
> Lin
> 
> 
> [1] https://github.com/mdavidsaver/ca2aa

Also note https://github.com/epicsdeb/carchivetools


References:
Questions about Archiver Appliance deployment Wang, Lin

Navigate by Date:
Prev: Questions about Archiver Appliance deployment Wang, Lin
Next: NDDriverStdArrays problem with configuration Marcin Lukaszewski
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  <20172018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Questions about Archiver Appliance deployment Wang, Lin
Next: NDDriverStdArrays problem with configuration Marcin Lukaszewski
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  <20172018  2019  2020  2021  2022  2023  2024