EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RE: NDArrayPool::alloc failing
From: Mark Rivers <[email protected]>
To: Phil Atkin <[email protected]>, "[email protected]" <[email protected]>
Date: Fri, 29 Jul 2016 13:29:15 +0000
Hi Phil,

>
Switch to second phase, then first call to processCallbacks

     *   Deallocate my memory buffer, which is no longer required
     *   Call NDArrayPool->release on my NDArray
     *   Call alloc again, asking for a much smaller array and with pData null

It seems to me like step 1) above is dangerous.  How do you know that some other plugin does not still have that NDArray allocated when you deallocate the memory buffer?  Plugins sometimes keep the last NDArray allocated until they receive the next one. Those plugins will now have an NDArray with an invalid memory pointer.

Ignoring the above problem, it seems like what are doing might work with the following 2 minor changes:

- When you deallocate your memory buffer also set NDArray->pData to 0.

Change line 642 in NDArrayPool.cpp as follows:

 -             free(freeArray->pData);
 +            if (freeArray->pData) free(freeArray->pData)

That change to NDArrayPool.cpp seems like a good safety check in any case.

> I don't quite understand the use-case for supplying pData if NDArrayPool is going to try to deallocate it at some point in the future.
> That timing of the deallocation is beyond the control of the code that allocates the buffer, so I would have thought that it was a recipe for problems;
> this is perhaps where my understanding is lacking.

You may be correct about that.  I just searched all of the existing areaDetector drivers in synApps, which includes all of those in areaDetector, and they never pass a non-zero value for pData.  It was probably something I thought could be useful when designing the NDArrayPool class, but has not been used in practice.

> My problem would be fixed if NDArrayPool::alloc did not ever try to free a memory block if the dataSize was set to zero.
> But that may break a behaviour on which others are relying.

My suggestion is to set pData to zero and then we don't call free() in that case.  But you still have the problem I mentioned at the beginning.

> It seems that my other option is that, when I have alloc'ed an NDArray to point to my buffer, and I destroy the buffer then I shut my eyes,
> reach inside the NDArray and set its pData member to zero.  Now, NDArrayPool has a valid, empty slot it can manage.
> Calling free on a nullptr is valid and does nothing.  I've tried this and it seems to avoid the crash, albeit while violating every known principle of
> software engineering.  Should I do anything else to make sure the array pool's contents are valid?

Yes, that was my idea above too.

> Can I be sure that, if I still hold a reference to a particular NDArray that I allocated for this purpose, that I've still got hold of the right one?

Yes, you can be sure of that.  However, I don't think you can be sure that other plugins don't also have it allocated.

We could add an NDArray::getReferenceCount method.  If you have the array reserved and getReferenceCount=1 then you can be sure all the plugins have released it.

Mark


________________________________
From: Phil Atkin [[email protected]]
Sent: Friday, July 29, 2016 7:56 AM
To: Mark Rivers; [email protected]
Subject: Re: NDArrayPool::alloc failing

Hi Mark, and thanks for the reply.

I think that the sequence is actually more like:

First phase, first call to processCallbacks

  1.  Allocate an NDArray with pData pointing to a memory buffer that I have allocated and manage myself
  2.  Call doCallbacksGenericPointer, passing my NDArray

First phase, subsequent calls to processCallbacks

  1.  Call NDArrayPool->release on my NDArray
  2.  Call alloc again, passing pData to my memory buffer
  3.  Call doCallbacksGenericPointer, passing my NDArray

(I've confirmed that in each case the same NDArray is re-used, and it carries on pointing to the buffer I've allocated)

Switch to second phase, then first call to processCallbacks

  1.  Deallocate my memory buffer, which is no longer required
  2.  Call NDArrayPool->release on my NDArray
  3.  Call alloc again, asking for a much smaller array and with pData null

It's this last step that causes a crash.  From what I can tell, the NDArray that's created by the initial call has its dataSize member set to _zero_, even though I correctly specified the size in the initial call to NDArrayPool::alloc.  That's kind of what I'd expect on general principles: since NDArrayPool isn't managing the memory, its size has zero impact on the pool's capacity.  But then I'm asking to actually allocate some memory at the switch to the second phase.  There's a free NDArray available but about 100 bytes are needed and its dataSize is zero - so NDArrayPool::alloc tries to free the pData, and it's already been freed.  So this is the crucial point, not that the memory is incompatible with free (actually, I don't even know if it's incompatible; I'm allocating with new[] ).

I don't quite understand the use-case for supplying pData if NDArrayPool is going to try to deallocate it at some point in the future.  That timing of the deallocation is beyond the control of the code that allocates the buffer, so I would have thought that it was a recipe for problems; this is perhaps where my understanding is lacking.
My problem would be fixed if NDArrayPool::alloc did not ever try to free a memory block if the dataSize was set to zero.  But that may break a behaviour on which others are relying.

It seems that my other option is that, when I have alloc'ed an NDArray to point to my buffer, and I destroy the buffer then I shut my eyes, reach inside the NDArray and set its pData member to zero.  Now, NDArrayPool has a valid, empty slot it can manage.  Calling free on a nullptr is valid and does nothing.  I've tried this and it seems to avoid the crash, albeit while violating every known principle of software engineering.  Should I do anything else to make sure the array pool's contents are valid?  Can I be sure that, if I still hold a reference to a particular NDArray that I allocated for this purpose, that I've still got hold of the right one?

I wish I could debug the NDxx code so I could tell you definitively what's it's doing, but I'm not sure how to achieve that.  (I'm using Visual Studio to debug my IOC/plugin, but I'm linking to a release build of ADBase.dll).

Sorry this has turned into such a long complicated story!  Cheers,

Phil

On 28/07/2016 23:52, Mark Rivers wrote:
Hi Phil,

I would like to understand the situation in which this could be happening.

It seems like you must be doing the following:


1.      Allocating an NDArray with pData pointing to a memory buffer that you have allocated and manage myself

2.      Returning that array to the pool with NDArrayPool->release()

3.      Allocating another array from the pool with pData null, but with a size that is larger than the one you allocated in step 1.

Is this the case?  If so, then I do see the problem.  It finds a free array in the pool, then finds that its size it not large enough, so it calls free() to free that memory and allocated new memory.  Because you used some other mechanism to allocate your memory it crashes.

This is a scenario I had not envisioned.  We could handle this by setting a flag that says that if the array was allocated with pData not null then do not call free().  In this case it would need to keep searching for another  existing NDArray that was large enough, and if none is found then allocated a new one.  This has 2 disadvantages:


-          It complicates the allocation process

-          It potentially leads to significant memory leaks.  The NDArray that you allocated may never be able to be used again because there are no requests that are small enough.

The simplest solution is just to document the restriction that if you pass a non-null pData it must have been allocated with some mechanism that is compatible with free().

Sinisa Veseli [email protected]<redir.aspx?REF=na4E1IF-vwrwfAOMtHWnk2ThCXJOFA1qKiHxCiR-xqDISSZnsLfTCAFtYWlsdG86c3Zlc2VsaUBhcHMuYW5sLmdvdg..> has a patch for NDArray that allows them to subclass it, because they want to store additional information in the NDArray.  Perhaps it would help with this problem.  I think not though, because this is really a problem with NDArrayPool and not NDArray itself.

Mark


From: [email protected]<redir.aspx?REF=u42VwISbQcoSkElB2GFnA0X3sytT70gtRiNzgiT1BR7ISSZnsLfTCAFtYWlsdG86dGVjaC10YWxrLWJvdW5jZXNAYXBzLmFubC5nb3Y.> [mailto:[email protected]<redir.aspx?REF=u42VwISbQcoSkElB2GFnA0X3sytT70gtRiNzgiT1BR7ISSZnsLfTCAFtYWlsdG86dGVjaC10YWxrLWJvdW5jZXNAYXBzLmFubC5nb3Y.>] On Behalf Of Phil Atkin
Sent: Thursday, July 28, 2016 12:14 PM
To: [email protected]<redir.aspx?REF=rjhKzqoCPXDKLtaIt7laiE_TbahJjgZmZPJe99eqBrXISSZnsLfTCAFtYWlsdG86dGVjaC10YWxrQGFwcy5hbmwuZ292>
Subject: NDArrayPool::alloc failing

In my areaDetector plugin I'm calling this method in one of two ways, depending on a PV value:

  *   With pData pointing to a memory buffer that I've allocated and manage myself;
  *   With pData null

All seems to work well when operated continuously in either mode, but I see a crash in RtlFreeHeap when I process at least one frame in the first mode, and then switch to the second.  This does not happen if I perform the same sequence of steps but instead of passing the pData to my buffer, I pass null and then copy from my buffer to the output - but this is a large buffer so I'd like to avoid this.

I haven't found anything in the documentation that says as much but I had assumed that, if I pass my own buffer to alloc, NDArrayPool would regard me as its owner and would never try to deallocate it.  (I would also expect it to exclude my buffer's size from its allocation limit calculations, but that's a secondary concern).  However, that appears not to be the case: NDArrayPool::alloc is deciding to free a now-unused NDArray that's pointing to my buffer and so it's calling free(freeArray->pData); that's failing because the memory was allocated in a manner incompatible with free.  NDArray seems to contain nothing that indicates who allocated the buffer, so NDArrayPool cannot reasonably alter its behaviour accordingly.

This behaviour doesn't seem right to me (on general principles) - but maybe I've misunderstood how I'm supposed to use this API?
Thanks,

Phil
--
[cid:[email protected]]Pixel Analytics is a limited company registered in England. Company number: 7747526; Registered office: 93A New Road, Haslingfield, Cambridge CB23 1LP

--
[cid:[email protected]]Pixel Analytics is a limited company registered in England. Company number: 7747526; Registered office: 93A New Road, Haslingfield, Cambridge CB23 1LP

JPEG image

PNG image


Replies:
Re: NDArrayPool::alloc failing Phil Atkin
Re: NDArrayPool::alloc failing Michael Davidsaver
References:
NDArrayPool::alloc failing Phil Atkin
RE: NDArrayPool::alloc failing Mark Rivers
Re: NDArrayPool::alloc failing Phil Atkin

Navigate by Date:
Prev: Re: Epics-Digitizer Communication Hulusi Öz
Next: Re: Epics-Digitizer Communication Eric Norum
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: NDArrayPool::alloc failing Phil Atkin
Next: Re: NDArrayPool::alloc failing Phil Atkin
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 29 Jul 2016 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·