EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RE: NDArrayPool::alloc failing
From: Mark Rivers <[email protected]>
To: "'Phil Atkin'" <[email protected]>, "[email protected]" <[email protected]>
Date: Thu, 28 Jul 2016 22:52:58 +0000

Hi Phil,

 

I would like to understand the situation in which this could be happening.

 

It seems like you must be doing the following:

 

1.      Allocating an NDArray with pData pointing to a memory buffer that you have allocated and manage myself

2.      Returning that array to the pool with NDArrayPool->release()

3.      Allocating another array from the pool with pData null, but with a size that is larger than the one you allocated in step 1.

 

Is this the case?  If so, then I do see the problem.  It finds a free array in the pool, then finds that its size it not large enough, so it calls free() to free that memory and allocated new memory.  Because you used some other mechanism to allocate your memory it crashes.

 

This is a scenario I had not envisioned.  We could handle this by setting a flag that says that if the array was allocated with pData not null then do not call free().  In this case it would need to keep searching for another  existing NDArray that was large enough, and if none is found then allocated a new one.  This has 2 disadvantages:

 

-          It complicates the allocation process

-          It potentially leads to significant memory leaks.  The NDArray that you allocated may never be able to be used again because there are no requests that are small enough.

 

The simplest solution is just to document the restriction that if you pass a non-null pData it must have been allocated with some mechanism that is compatible with free().

 

Sinisa Veseli [email protected] has a patch for NDArray that allows them to subclass it, because they want to store additional information in the NDArray.  Perhaps it would help with this problem.  I think not though, because this is really a problem with NDArrayPool and not NDArray itself.

 

Mark

 

 

From: [email protected] [mailto:[email protected]] On Behalf Of Phil Atkin
Sent: Thursday, July 28, 2016 12:14 PM
To: [email protected]
Subject: NDArrayPool::alloc failing

 

In my areaDetector plugin I'm calling this method in one of two ways, depending on a PV value:

  • With pData pointing to a memory buffer that I've allocated and manage myself;
  • With pData null

All seems to work well when operated continuously in either mode, but I see a crash in RtlFreeHeap when I process at least one frame in the first mode, and then switch to the second.  This does not happen if I perform the same sequence of steps but instead of passing the pData to my buffer, I pass null and then copy from my buffer to the output - but this is a large buffer so I'd like to avoid this.

I haven't found anything in the documentation that says as much but I had assumed that, if I pass my own buffer to alloc, NDArrayPool would regard me as its owner and would never try to deallocate it.  (I would also expect it to exclude my buffer's size from its allocation limit calculations, but that's a secondary concern).  However, that appears not to be the case: NDArrayPool::alloc is deciding to free a now-unused NDArray that's pointing to my buffer and so it's calling free(freeArray->pData); that's failing because the memory was allocated in a manner incompatible with free.  NDArray seems to contain nothing that indicates who allocated the buffer, so NDArrayPool cannot reasonably alter its behaviour accordingly.

This behaviour doesn't seem right to me (on general principles) - but maybe I've misunderstood how I'm supposed to use this API?

Thanks,

Phil

--
Pixel Analytics is a limited company registered in England. Company number: 7747526; Registered office: 93A New Road, Haslingfield, Cambridge CB23 1LP


Replies:
Re: NDArrayPool::alloc failing Phil Atkin
References:
NDArrayPool::alloc failing Phil Atkin

Navigate by Date:
Prev: RE: may I start a pcaspy ioc together with a streamdevice ioc? Mark Rivers
Next: Re: Epics-Digitizer Communication Ford, Christopher
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: NDArrayPool::alloc failing Phil Atkin
Next: Re: NDArrayPool::alloc failing Phil Atkin
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 29 Jul 2016 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·