Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014  2015  2016  2017  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014  2015  2016  2017 
<== Date ==> <== Thread ==>

Subject: Re: JCA problems and questions
From: Matej Sekoranja <matej.sekoranja@cosylab.com>
To: "Hu, Yong" <yhu@bnl.gov>
Cc: "tech-talk@aps.anl.gov" <tech-talk@aps.anl.gov>
Date: Thu, 25 Oct 2012 11:20:36 +0200
Hi,

I’d like to ask some questions related to TCP connection and automatic resizing of  max_array_bytes as well as show some of my thoughts.

 

1.  “100 connection * 10MB = 1GB”: This means the client is connected with 100 different Channel Access Servers (IOCs).

 

1) Suppose this client is CSS running on Linux.  How can we find out how many CA TCP connections are opened on the Linux machine? Is there a utility / plug-in in the CSS to provide some connection info just like the IOC shell command ‘casr’ which can report detailed info about the TCP connections on the CA Server side?


No API on the CA client side.
For CAJ you could do something like:

((CAJContext)context).getTransportRegistry().numberOfActiveTransports() to get number of all active TCP connections.
 

 

2) In this case, EPICS_CA_MAX_ARRAY_BYTES is set to 10MB on the CA client side. I did several tests to see the memory used by several clients “camonitor” on a Linux machine. The IOC (CA server) is one softIoc with some array-type records such as compress, waveform, subArray and aSub which have data size a little bit bigger than 16384 Bytes so that EPICS_CA_MAX_ARRAY_BYTES is set to 1 MB on the IOC side. I opened three terminals. One terminal is used to camonitor a scalar PV “camonitor yhuHost:aiExample”, one is used for “camonitor array-type-record” and another terminal is for the “top” and “free -m” commands to monitor the memory usage. The result from the “top” command seems interesting: the value of VIRT/RES for either “camonitor yhuHost:aiExample” or “camonitor array-type-record” is something like 11m / 2680 which means the virtual memory is around 10MB, but the actually used resident memory is much smaller. The “free -m” command shows the same thing: the used memory by those two “camonitor” is actually very small, not 2 connection * 10MB=20MB. So, my understanding of “100 connection * 10MB = 1GB” is that 1 GB is the total max. virtual memory size, but the actual physical memory consumed by those TCP connections,  which depends on the data size of array-type records on the IOCs’ sides, is usually less and smaller than 1GB.


Actual size of the client receive buffer is manager internally.
The old version initially reserved EPICS_CA_MAX_ARRAY_BYTES, but later do not.
I think C version uses initial size of 64k and if this "limit" is reached it resizes to EPICS_CA_MAX_ARRAY_BYTES.
The server might behave differently due to real-time constraints (reallocation is not always a good idea).

Matej
 

Please note that my test setup is several CA clients connected to only one CA server, which seems reverse to the setup “100 connection * 10MB = 1GB” where only one CA client is connected to several CA servers. But I think these two setups should produce almost the same effects / results for testing memory usage by CA TCP connections.

 

2. “automatically resized up to max_array_bytes”: I agree with Matej that this is very convenient on the client side. I guess most of us have experienced having to setup EPICS_CA_MAX_ARRAY_BYTES to a bigger number on both IOC side (in st.cmd) and OPI side (i.e. CSS, EDM) when we deal with array-like records such as compress, waveform, subArray, aSub, etc. It would be nice to automatically (or semi-automatically) setup EPICS_CA_MAX_ARRAY_BYTES to an appropriate number on the IOC side during iocInit. I guess it might be feasible and here is what I think:

 

1) add something like iterateRecords(calculateMaxArrayBytes, NULL) at initDatabase() during iocBuild() to calculate the max. memory size (number of bytes) that an array-type record needs. For instance, if all records in the IOC are scalar-type,  we will do nothing with EPICS_CA_MAX_ARRAY_BYTES for this case. If there is a waveform record in the IOC, we can calculate the max. memory size (say max_array_mem_size) allocated for the waveform record by prec->nelm * dbValueSize(prec->FTVL). Use similar algorithms for any other array-type records to get their max. memory sizes and eventually we can get the max. value of max_array_mem_size. If (max_array_mem_size * 1.2)> 16384,  then we setup EPICS_CA_MAX_ARRAY_BYTES = max_array_mem_size * 1.2. The coefficient “1.2” (or other value) is for adding up additional bytes for compound data types (i.e. DBR_GR_DOUBLE).

 

2) the algorithm described above won’t work if the IOC contains non-standard (not included in the base) records such as records from the synApps package. In this case, we have to manually configure EPICS_CA_MAX_ARRAY_BYTES in the IOC st.cmd file. EPICS_CA_MAX_ARRAY_BYTES is defined before iocInit() so that iterateRecords(calculateMaxArrayBytes, NULL) just returns, does nothing.

 

3) So, iterateRecords(calculateMaxArrayBytes, NULL) will make EPICS_CA_MAX_ARRAY_BYTES transparent on the IOC side for most users in most cases. But manual configuration of  EPICS_CA_MAX_ARRAY_BYTES will eventually work for all cases.

 

Sorry for this long message. If most of us think it is worth implementing this kind of semi-automatic configuration of  EPICS_CA_MAX_ARRAY_BYTES on the IOC side, I will give a try when I have extra time.

 

Thanks,

 

Yong

 

 

From: tech-talk-bounces@aps.anl.gov [mailto:tech-talk-bounces@aps.anl.gov] On Behalf Of Matej Sekoranja


Sent: Tuesday, October 23, 2012 4:33 PM
To: tech-talk@aps.anl.gov
Subject: Re: JCA problems and questions

 

Hi,

 

Rok Sabjan notified me about this thread. Thanks to Lewis for replies.

 

The old send buffer algorithm was to initialize the send buffer size to max_array_bytes and

automatically resize on demand (there is one send buffer per TCP connection). Not something

one would dare to use on a server, however very convenient on the client side.

 

However, if a client has a lot of connections there is a lot of memory required when max_array_bytes is large (e.g. 100 connection * 10MB = 1GB!).

Current algorithm starts with an initial size of 1k that can be automatically resized up to max_array_bytes.

This also mimics C++ CA algorithm (that has also evolved over the years).

 

Cheers,

Matej



References:
Re: JCA problems and questions Matej Sekoranja
RE: JCA problems and questions Hu, Yong

Navigate by Date:
Prev: Re: JCA problems and questions Matej Sekoranja
Next: RE: JCA problems and questions Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014  2015  2016  2017 
Navigate by Thread:
Prev: RE: JCA problems and questions Hu, Yong
Next: ASKAP Opening Ceremony webcast Juan.Guzman
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  <20122013  2014  2015  2016  2017 
ANJ, 18 Nov 2013 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·