EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: RE: NDArrayPool::alloc failing
From: Mark Rivers <[email protected]>
To: "'Phil Atkin'" <[email protected]>, "[email protected]" <[email protected]>
Date: Fri, 12 Aug 2016 17:59:03 +0000

Hi Phil,

 

Ø  Please correct me if I'm wrong:

Ø  If a plugin downstream of mine has 'Callbacks block' set, then there can be no queue of arrays and the downstream plugin will execute after my plugin, so my plugin will never update the array while the downstream plugin is executing.

 

That is correct.

 

Ø  There is no way to force the downstream plugin to be configured with 'Callbacks block' - because the BlockingCallbacks PV can always be set to false at runtime.  But I can set the initial state of this to true in the startup script.

 

You can it to true in the startup script.  After iocInit in the startup script you could also do “dbpf” commands to the BlockingCallbacks PV .SDIS and .DISV fields to disable that record so users cannot change it.

 

Ø  Finally, is there an easy way to force my own plugin to run with blocking callbacks?

 

You could just use the answer to the previous question.  You could probably also implement the virtual method NDPluginDriver::driverCallback.  That should give you complete control over whether arrays are queued or not.  I have never tested this, I have only ever used the base class implementation.

 

Mark

 

 

From: Phil Atkin [mailto:[email protected]]
Sent: Friday, August 12, 2016 11:43 AM
To: Mark Rivers; [email protected]
Subject: Re: NDArrayPool::alloc failing

 

Mark,

Please correct me if I'm wrong:

If a plugin downstream of mine has 'Callbacks block' set, then there can be no queue of arrays and the downstream plugin will execute after my plugin, so my plugin will never update the array while the downstream plugin is executing.

There is no way to force the downstream plugin to be configured with 'Callbacks block' - because the BlockingCallbacks PV can always be set to false at runtime.  But I can set the initial state of this to true in the startup script.

Finally, is there an easy way to force my own plugin to run with blocking callbacks?

Cheers,

Phil

 

On 29/07/2016 17:29, Mark Rivers wrote:

We could add an NDArray::getReferenceCount method.  If you have the array reserved and getReferenceCount=1 then you can be sure all the plugins have released it.

 

What if they haven't?  Can I wait, knowing that they will eventually?

 

No, the reference count may never go to zero.  As I said many plugins do not call pArray->release() on their most recent callback array until they get the next callback.  This allows them to implement pAsynGenericPointer->read() to return the last array they received.  But if that plugin is never called again (user disabled it) then that array is never released.

 

A different approach would be to switch to getting NDArrayPool to do the allocation for me, then wrangle my code so that I can use that buffer instead.  Then I need to make sure that my plugin always uses that same buffer (across multiple calls to processCallbacks) because its contents gradually accumulate over multiple calls.  Is that as simple as not calling release on it during every call to processCallbacks as seems to be the norm, but only when I no longer need it?

 

That should not crash, but it won’t really give the desired behavior.  For example, imagine a file writing plugin that is receiving callbacks, but is not able to write quite as fast as the arrays are arriving.  The NDArrays go into a queue.  The expectation is that each file written will contain the data when the driver wrote that array.  But if your driver is using the same buffer and changing it then the data buffer will be updating while the arrays are in the queue, which is not what the user expects.

 

Mark

 

 

 

From: Phil Atkin [mailto:[email protected]]
Sent: Friday, July 29, 2016 9:17 AM
To: Mark Rivers; [email protected]
Subject: Re: NDArrayPool::alloc failing

 

Hi Mark,

On 29/07/2016 14:29, Mark Rivers wrote:

Switch to second phase, then first call to processCallbacks
 
     *   Deallocate my memory buffer, which is no longer required
     *   Call NDArrayPool->release on my NDArray
     *   Call alloc again, asking for a much smaller array and with pData null
 
It seems to me like step 1) above is dangerous.  How do you know that some other plugin does not still have that NDArray allocated when you deallocate the memory buffer?  Plugins sometimes keep the last NDArray allocated until they receive the next one. Those plugins will now have an NDArray with an invalid memory pointer.

That's a fair point, but I'm not sure what to do about it.

Ignoring the above problem, it seems like what are doing might work with the following 2 minor changes:
 
- When you deallocate your memory buffer also set NDArray->pData to 0.

I find this is sufficient to prevent the crash - at least in the circumstances in which it was failing before.

Change line 642 in NDArrayPool.cpp as follows:
 
 -             free(freeArray->pData);
 +            if (freeArray->pData) free(freeArray->pData)
 
That change to NDArrayPool.cpp seems like a good safety check in any case.

As I mentioned, at least on the Windows platform it's safe to call free on a null pointer, and it does nothing.


We could add an NDArray::getReferenceCount method.  If you have the array reserved and getReferenceCount=1 then you can be sure all the plugins have released it.

What if they haven't?  Can I wait, knowing that they will eventually?


 

A different approach would be to switch to getting NDArrayPool to do the allocation for me, then wrangle my code so that I can use that buffer instead.  Then I need to make sure that my plugin always uses that same buffer (across multiple calls to processCallbacks) because its contents gradually accumulate over multiple calls.  Is that as simple as not calling release on it during every call to processCallbacks as seems to be the norm, but only when I no longer need it?

Phil


 
________________________________
From: Phil Atkin [[email protected]]
Sent: Friday, July 29, 2016 7:56 AM
To: Mark Rivers; [email protected]
Subject: Re: NDArrayPool::alloc failing
 
Hi Mark, and thanks for the reply.
 
I think that the sequence is actually more like:
 
First phase, first call to processCallbacks
 
  1.  Allocate an NDArray with pData pointing to a memory buffer that I have allocated and manage myself
  2.  Call doCallbacksGenericPointer, passing my NDArray
 
First phase, subsequent calls to processCallbacks
 
  1.  Call NDArrayPool->release on my NDArray
  2.  Call alloc again, passing pData to my memory buffer
  3.  Call doCallbacksGenericPointer, passing my NDArray
 
(I've confirmed that in each case the same NDArray is re-used, and it carries on pointing to the buffer I've allocated)
 
Switch to second phase, then first call to processCallbacks
 
  1.  Deallocate my memory buffer, which is no longer required
  2.  Call NDArrayPool->release on my NDArray
  3.  Call alloc again, asking for a much smaller array and with pData null
 
It's this last step that causes a crash.  From what I can tell, the NDArray that's created by the initial call has its dataSize member set to _zero_, even though I correctly specified the size in the initial call to NDArrayPool::alloc.  That's kind of what I'd expect on general principles: since NDArrayPool isn't managing the memory, its size has zero impact on the pool's capacity.  But then I'm asking to actually allocate some memory at the switch to the second phase.  There's a free NDArray available but about 100 bytes are needed and its dataSize is zero - so NDArrayPool::alloc tries to free the pData, and it's already been freed.  So this is the crucial point, not that the memory is incompatible with free (actually, I don't even know if it's incompatible; I'm allocating with new[] ).
 
I don't quite understand the use-case for supplying pData if NDArrayPool is going to try to deallocate it at some point in the future.  That timing of the deallocation is beyond the control of the code that allocates the buffer, so I would have thought that it was a recipe for problems; this is perhaps where my understanding is lacking.
My problem would be fixed if NDArrayPool::alloc did not ever try to free a memory block if the dataSize was set to zero.  But that may break a behaviour on which others are relying.
 
It seems that my other option is that, when I have alloc'ed an NDArray to point to my buffer, and I destroy the buffer then I shut my eyes, reach inside the NDArray and set its pData member to zero.  Now, NDArrayPool has a valid, empty slot it can manage.  Calling free on a nullptr is valid and does nothing.  I've tried this and it seems to avoid the crash, albeit while violating every known principle of software engineering.  Should I do anything else to make sure the array pool's contents are valid?  Can I be sure that, if I still hold a reference to a particular NDArray that I allocated for this purpose, that I've still got hold of the right one?
 
I wish I could debug the NDxx code so I could tell you definitively what's it's doing, but I'm not sure how to achieve that.  (I'm using Visual Studio to debug my IOC/plugin, but I'm linking to a release build of ADBase.dll).
 
Sorry this has turned into such a long complicated story!  Cheers,
 
Phil
 
On 28/07/2016 23:52, Mark Rivers wrote:
Hi Phil,
 
I would like to understand the situation in which this could be happening.
 
It seems like you must be doing the following:
 
 
1.      Allocating an NDArray with pData pointing to a memory buffer that you have allocated and manage myself
 
2.      Returning that array to the pool with NDArrayPool->release()
 
3.      Allocating another array from the pool with pData null, but with a size that is larger than the one you allocated in step 1.
 
Is this the case?  If so, then I do see the problem.  It finds a free array in the pool, then finds that its size it not large enough, so it calls free() to free that memory and allocated new memory.  Because you used some other mechanism to allocate your memory it crashes.
 
This is a scenario I had not envisioned.  We could handle this by setting a flag that says that if the array was allocated with pData not null then do not call free().  In this case it would need to keep searching for another  existing NDArray that was large enough, and if none is found then allocated a new one.  This has 2 disadvantages:
 
 
-          It complicates the allocation process
 
-          It potentially leads to significant memory leaks.  The NDArray that you allocated may never be able to be used again because there are no requests that are small enough.
 
The simplest solution is just to document the restriction that if you pass a non-null pData it must have been allocated with some mechanism that is compatible with free().
 
Sinisa Veseli [email protected]<redir.aspx?REF=na4E1IF-vwrwfAOMtHWnk2ThCXJOFA1qKiHxCiR-xqDISSZnsLfTCAFtYWlsdG86c3Zlc2VsaUBhcHMuYW5sLmdvdg..> has a patch for NDArray that allows them to subclass it, because they want to store additional information in the NDArray.  Perhaps it would help with this problem.  I think not though, because this is really a problem with NDArrayPool and not NDArray itself.
 
Mark
 
 
From: [email protected]<redir.aspx?REF=u42VwISbQcoSkElB2GFnA0X3sytT70gtRiNzgiT1BR7ISSZnsLfTCAFtYWlsdG86dGVjaC10YWxrLWJvdW5jZXNAYXBzLmFubC5nb3Y.> [mailto:[email protected]<redir.aspx?REF=u42VwISbQcoSkElB2GFnA0X3sytT70gtRiNzgiT1BR7ISSZnsLfTCAFtYWlsdG86dGVjaC10YWxrLWJvdW5jZXNAYXBzLmFubC5nb3Y.>] On Behalf Of Phil Atkin
Sent: Thursday, July 28, 2016 12:14 PM
To: [email protected]<redir.aspx?REF=rjhKzqoCPXDKLtaIt7laiE_TbahJjgZmZPJe99eqBrXISSZnsLfTCAFtYWlsdG86dGVjaC10YWxrQGFwcy5hbmwuZ292>
Subject: NDArrayPool::alloc failing
 
In my areaDetector plugin I'm calling this method in one of two ways, depending on a PV value:
 
  *   With pData pointing to a memory buffer that I've allocated and manage myself;
  *   With pData null
 
All seems to work well when operated continuously in either mode, but I see a crash in RtlFreeHeap when I process at least one frame in the first mode, and then switch to the second.  This does not happen if I perform the same sequence of steps but instead of passing the pData to my buffer, I pass null and then copy from my buffer to the output - but this is a large buffer so I'd like to avoid this.
 
I haven't found anything in the documentation that says as much but I had assumed that, if I pass my own buffer to alloc, NDArrayPool would regard me as its owner and would never try to deallocate it.  (I would also expect it to exclude my buffer's size from its allocation limit calculations, but that's a secondary concern).  However, that appears not to be the case: NDArrayPool::alloc is deciding to free a now-unused NDArray that's pointing to my buffer and so it's calling free(freeArray->pData); that's failing because the memory was allocated in a manner incompatible with free.  NDArray seems to contain nothing that indicates who allocated the buffer, so NDArrayPool cannot reasonably alter its behaviour accordingly.
 
This behaviour doesn't seem right to me (on general principles) - but maybe I've misunderstood how I'm supposed to use this API?
Thanks,
 
Phil
--
[cid:[email protected]]Pixel Analytics is a limited company registered in England. Company number: 7747526; Registered office: 93A New Road, Haslingfield, Cambridge CB23 1LP
 
--
[cid:[email protected]]Pixel Analytics is a limited company registered in England. Company number: 7747526; Registered office: 93A New Road, Haslingfield, Cambridge CB23 1LP
 

 

--
Pixel Analytics is a limited company registered in England. Company number: 7747526; Registered office: 93A New Road, Haslingfield, Cambridge CB23 1LP

 

--
Pixel Analytics is a limited company registered in England. Company number: 7747526; Registered office: 93A New Road, Haslingfield, Cambridge CB23 1LP


References:
NDArrayPool::alloc failing Phil Atkin
RE: NDArrayPool::alloc failing Mark Rivers
Re: NDArrayPool::alloc failing Phil Atkin
RE: NDArrayPool::alloc failing Mark Rivers
Re: NDArrayPool::alloc failing Phil Atkin
RE: NDArrayPool::alloc failing Mark Rivers
Re: NDArrayPool::alloc failing Phil Atkin

Navigate by Date:
Prev: RE: wait/dealy inside a record Mark Rivers
Next: Is it possible for channel access to send duplicates? Zhang, Dehong
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: NDArrayPool::alloc failing Phil Atkin
Next: Re: NDArrayPool::alloc failing Michael Davidsaver
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 12 Aug 2016 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·