EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: archive engine
From: "Kasemir, Kay" <[email protected]>
To: Caizhen Liu <[email protected]>, "[email protected]" <[email protected]>
Date: Fri, 14 Oct 2016 15:22:58 +0000
Hi:

>We have about 4000 PVs being archived at the moment.
> It seems that the archiving have some issue starting from PV “OPS-EPICS:R:CCG1”.
> The queue is full and overruns happens. And there is no data record for PV OPS-EPICS:R:CCG1 in the postgres  database.
> When I reduced the number of channels to be 90, everything seems ok.

More important than the number of PVs or the specific PV names is the number of samples that these channels produce.
The main web page of the archive engine will show you a "write count" and a "write duration".
From watching that for a while, you can compute the number of values that the engine can write to your relational database.

Bottom line is that the RDB approach for archiving is very reliable, but not fast.
You may simply reach some maximum supported number of samples/sec that you can write to your database.

When you build CSS from sources, you can execute the RDBArchiveWriterTest (https://github.com/ControlSystemStudio/cs-studio/blob/master/applications/archive/archive-plugins/org.csstudio.archive.writer.test/test/org/csstudio/archive/writer/rdb/RDBArchiveWriterTest.java).
Way back, I got these types of performance readings:
PostgreSQL: ~12000 samples/sec
MySQL: ~21000 samples/sec

Those are write speeds when the database is on the same computer that's also running the test, and nothing else is using the RDB.
In the real world, when you connect to a remote database that's also doing its day job, the numbers a lower.

The EPICS Archiver Appliance, https://slacmshankar.github.io/epicsarchiver_docs/index.html, is faster because it uses its own set of files instead of a relational database.

-Kay

Replies:
RE: archive engine Caizhen Liu
RE: archive engine Caizhen Liu
References:
FW: archive engine Caizhen Liu

Navigate by Date:
Prev: 答复: Question about epics and pcaspy running in one computer lzf neu
Next: Re: Question about epics and pcaspy running in one computer Kasemir, Kay
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: FW: archive engine Caizhen Liu
Next: RE: archive engine Caizhen Liu
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 14 Oct 2016 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·