Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  <20102011  2012  2013  2014  2015  2016  2017  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  <20102011  2012  2013  2014  2015  2016  2017 
<== Date ==> <== Thread ==>

Subject: Re: labCA and EPICS_CA_MAX_ARRAY_BYTES
From: Andrew Johnson <anj@aps.anl.gov>
To: tech-talk@aps.anl.gov
Date: Fri, 28 May 2010 06:04:02 -0500
Hi Matt,

On Thursday 27 May 2010 20:25:01 Matt Newville wrote:
>
> Yes, I understand this.  Obviously, we're talking about transferring
> data arrays which already exist on real IOCs, on networks that can
> transfer them.  The advice of "increase EPICS_CA_MAX_ARRAY_BYTES and
> everything will work fine" implies (perhaps even demonstrates) that
> the default value of EPICS_CA_MAX_ARRAY_BYTES is too small.

The CA server gets one chance to make a complete copy of each array as part of 
the process of sending it to its clients, so it must allocate a buffer big 
enough to hold that array.  When a new client connects the server allocates 
two buffers of 16K each to that client (for send and receive), but when a 
request comes in for something bigger it attempts to upgrade the relevant 
buffer to MAX_ARRAY_BYTES plus a bit more to cover the header overhead.  Some 
of our vxWorks IOCs at the APS can (and do) have up to 250 clients, so they 
need 250*2*16K ~= 8MB of RAM just for default-sized CA buffers.  That's not a 
problem on a workstation class machine where the client count could be even 
larger if necessary, but some of our vxWorks IOCs have only about 32MB of RAM 
total.

> > This is especially important for servers, since IOCs may have limited
> > memory and high availability requirements.
>
> Are there IOCs that cannot have 16k of PV data values?  The value
> cannot be set lower than 16k, so the limited-memory argument is a bit
> suspect.  Anyway, the issue that keeps cropping up is not whether IOCs
> can generate and serve the data, but whether clients can receive them.

The ColdFire IOCs that APS is using for BPMs and other FPGA-implemented 
processing have 16MB of RAM.  They obviously can't support as many clients as 
the 250 I mentioned above, and in many cases they also have one or more 
waveform records holding large acquisition history data buffers, so they have 
to have MAX_ARRAY_BYTES set to something larger than the default.

The standard APS Unix and Linux workstation login scripts have the environment 
variable EPICS_CA_MAX_ARRAY_BYTES set to 140000 which we found to be suitable 
for our systems.  Could you adopt a similar approach for your client machines?

> Is there really no technical reason for this value?  If it is only
> historical, and it can be changed with a one-line fix to
> base/configure/CONFIG_ENV, why not change it?

16K seems to be about right for a minimum buffer size from experience, I don't 
recall anyone complaining that it was too big, although before Jeff added the 
MAX_ARRAY_BYTES support it was also the maximum buffer size and that caused 
many complaints.  If you feel the default should be smaller I'm sure Jeff Hill 
will be happy to discuss that point with you (but not until he gets back to 
LANL, as he can't check his email while he's over here in Europe).

> It appears to me that
> the value is really used only in base/src/ca/cac.cpp (starting around
> line 185), and there only to set a maximum data size.  There, you'll
> see two interesting pieces of information:
>   a) setting EPICS_CA_MAX_ARRAY_BYTES < 16384 (MAX_TCP) will result in
> it being reset to 16384:
>       so that one cannot set this to a number smaller than 16384  (so
> much for allowing one to conserve
>       resources on memory-starved systems!!)
>   b) room for header information is added onto this value (apparently
> contradicting an earlier description),
>       up to 2*32 bytes.

Yes, we were getting complaints from people who didn't read the documentation, 
and there's only so many times you can write RTFM in email replies, so Jeff 
added the header size so the name reflects reality.

> > So that configuration option was meant to *allow* for larger arrays,
> > never intended to *restrict* the array size.
>
> But restricting is exactly the effect that it does have for client
> programs! The array already exists in some IOC, which is prepared to
> transfer the data to any and all clients.  But unless a client
> application knows how large an array is **BEFORE THE APPLICATION
> BEGINS USING CA**, it cannot get the data.

Or you just have some default environment variable (or default set in 
CONFIG_SITE_ENV) for all workstation users which is big enough for everything 
you think they'll need.

HTH,

- Andrew
-- 
The best FOSS code is written to be read by other humans -- Harald Welte


References:
labCA and EPICS_CA_MAX_ARRAY_BYTES John Dobbins
Re: labCA and EPICS_CA_MAX_ARRAY_BYTES Ralph Lange
Re: labCA and EPICS_CA_MAX_ARRAY_BYTES Matt Newville

Navigate by Date:
Prev: Re: EPICS wiki is down Andrew Johnson
Next: Re: labCA and EPICS_CA_MAX_ARRAY_BYTES Ralph Lange
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  <20102011  2012  2013  2014  2015  2016  2017 
Navigate by Thread:
Prev: Re: labCA and EPICS_CA_MAX_ARRAY_BYTES Matt Newville
Next: Re: labCA and EPICS_CA_MAX_ARRAY_BYTES Ralph Lange
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  <20102011  2012  2013  2014  2015  2016  2017 
ANJ, 02 Sep 2010 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·