Subject: |
Re: How to write a python-based backpup tool? |
From: |
Matt Newville <[email protected]> |
To: |
EPICS tech-talk <[email protected]> |
Date: |
Fri, 16 Sep 2011 16:30:03 -0500 |
Hi Emmanuel,
On Fri, Sep 16, 2011 at 3:09 PM, <[email protected]> wrote:
>> pvlist = []
>> for pvName in pvNames:
>> pvlist.append( epics.PV(pvName) )
>>
>> for pvs in pvlist:
>> val = pv.get()
>> <store value>
>>
>> With this sort of approach, I typically see on the order of 10ms per
>> PV connection on startup. That is, if I create and connect to 5K
>> PVs, it takes ~50 seconds (meaning 40 to 80 seconds) to get initial
>> values. I believe that is mostly the CA library, not the python part,
>> and I believe it would scale, suggesting that any save/restore process
>> that runs once and then quit would take 10 minutes for 50K PVs.
>
> Are you saying that 10 min for 50K PVs is also what is observed with C utilities?
> 10 min for 50K is rather long ...
I'd be happy to be shown wrong, and I'm certainly not the right person
here to answer for sure, but I think this might be the case.
> If that is the case, then one whould be careful to backup only PV which are of interest
> and not the 'entire' machine.
> How is this being managed at large installation?
I think they break up the set of variables across
processes/machines/NICs and/or have a long-running process that
repeatedly saves variables instead of
start process; connect to 50K PVs; write values; end process.
> I understand your points, but somehow I am having a hard time believing this.
> Did anyone benchmark their backup platform?
> (I remember seeing a few powerpoints on this issue a while back...)
For what it's worth, and also gladly proven wrong, etc: I recall that
with non-preemptive-callback context, creating and connecting PVs took
exactly 30ms, to very high precision and reproducibility. I never
fully understood that.... From that view, the 10 ms average with the
preemptive-context is an improvement.
It's probably easy to tell whether it is closer to 1, 10, or 30 ms
even with a few hundred PVs. But even at 1ms / PV connection, you
would probably want to not start a new connection per PV for saving
50K PVs.
> PS:
> Is there a difference between a connection to 50K Pvs versus, 50K connections to the same PVs?
Yes, these are definitely different.
--Matt
- Replies:
- Re: How to write a python-based backpup tool? emmanuel_mayssat
- Re: How to write a python-based backpup tool? Ralph Lange
- References:
- How to write a python-based backpup tool? emmanuel_mayssat
- Re: How to write a python-based backpup tool? Matt Newville
- Re: How to write a python-based backpup tool? emmanuel_mayssat
- Navigate by Date:
- Prev:
Re: How to write a python-based backpup tool? emmanuel_mayssat
- Next:
Re: How to write a python-based backpup tool? emmanuel_mayssat
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
<2011>
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
- Navigate by Thread:
- Prev:
Re: How to write a python-based backpup tool? emmanuel_mayssat
- Next:
Re: How to write a python-based backpup tool? emmanuel_mayssat
- Index:
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
<2011>
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
|