Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  <2017 Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  <2017
<== Date ==> <== Thread ==>

Subject: Re: areaDetector and HDF5 file saving segafaults
From: Hinko Kocevar <hinkocevar@gmail.com>
To: Mark Rivers <rivers@cars.uchicago.edu>
Cc: "tech-talk@aps.anl.gov" <tech-talk@aps.anl.gov>
Date: Fri, 17 Nov 2017 16:29:42 +0100
Maybe it helps, after yet another crash after longer time the
attribute in question is the same:

#2  0x00007ffff776bff7 in NDFileHDF5AttributeDataset::createHDF5Dataset (
    this=this@entry=0x7ffb2ffb2e10) at ../NDFileHDF5AttributeDataset.cpp:99
99   H5Pset_fill_value(cparm_, datatype_, ptrFillValue_);
(gdb) p *this
$3 = {
  _vptr.NDFileHDF5AttributeDataset = 0x7ffff799c5b0 <vtable for
NDFileHDF5AttributeDataset+16>, name_ = "CameraModel", dsetName_ =
"CameraModel", file_ = 72057594037982897,
  type_ = NDAttrString, groupName_ = "/entry/instrument/NDAttributes",
dataset_ = -1,
  dataspace_ = -1, memspace_ = -1, datatype_ = 216172782122083269,
  cparm_ = 720575940385270143, filespace_ = -1, ptrFillValue_ = 0x7ffb2fff4f30,
  dims_ = 0x7ffb1c058030, offset_ = 0x7ffb1c0c3cb0, chunk_ = 0x7ffb2ff89a80,
  maxdims_ = 0x7ffb2feedf00, virtualdims_ = 0x0, elementSize_ = 0x7ffb1c0c3cd0,
  isUndefined_ = false, rank_ = 1, nextRecord_ = 0, extraDimensions_ = 0,
  whenToSave_ = hdf5::OnFrame}
(gdb)

/Hinko


On Fri, Nov 17, 2017 at 3:42 PM, Hinko Kocevar <hinkocevar@gmail.com> wrote:
> Hi Mark,
>
> I finally got the machine and time to debug further.
>
> On Tue, Nov 14, 2017 at 1:15 PM, Mark Rivers <rivers@cars.uchicago.edu> wrote:
>> Hi Hinko,
>>
>>
>> Can you also print the contents of "this" in NDFileHDF5AttributeDataset::createHDF5Dataset()?  That will tell us the name of the attribute, which may be helpful.
>>
>
> Here it is:
>
> (gdb) frame 2
> #2  0x00007ffff776bff7 in NDFileHDF5AttributeDataset::createHDF5Dataset (
>     this=this@entry=0x7ffecc07bcb0) at ../NDFileHDF5AttributeDataset.cpp:99
> 99   H5Pset_fill_value(cparm_, datatype_, ptrFillValue_);
> (gdb) p this
> $1 = (NDFileHDF5AttributeDataset * const) 0x7ffecc07bcb0
> (gdb) p *this
> $2 = {
>   _vptr.NDFileHDF5AttributeDataset = 0x7ffff799c5b0 <vtable for
> NDFileHDF5AttributeDataset+16>, name_ = "CameraModel", dsetName_ =
> "CameraModel", file_ = 72057594037929019,
>   type_ = NDAttrString, groupName_ = "/entry/instrument/NDAttributes",
> dataset_ = -1,
>   dataspace_ = -1, memspace_ = -1, datatype_ = 216172782113947691,
>   cparm_ = 720575940379397441, filespace_ = -1, ptrFillValue_ = 0x7ffed7ffcfd0,
>   dims_ = 0x7ffed7ed2180, offset_ = 0x7ffecc0611d0, chunk_ = 0x7ffecc011ea0,
>   maxdims_ = 0x7ffed7d5f2e0, virtualdims_ = 0x0, elementSize_ = 0x7ffecc0611f0,
>   isUndefined_ = false, rank_ = 1, nextRecord_ = 0, extraDimensions_ = 0,
>   whenToSave_ = hdf5::OnFrame}
> (gdb)
>
>
>>
>> Can you also provide some more information on the crash.
>>
>> - Does this occur each time the file is written, or only once in a while?
>
> No, not each time.
> This time, it took less than a minute @ 10 files per second.
>
>>
>> - If once in a while what is the frequency?  1%, .1%, etc.?
>
> Now I'm running again and I have already saved >5000 files and still no crash..
>
>> - Can you send tell me the XML file you are using to define the NDAttributes, and for each of those that are PVAttributes also tell me the EPICS record type?
>
> You can find the file attached.
>
> Thanks,
> Hinko
>
>>
>> Mark
>>
>>
>> ________________________________
>> From: Hinko Kocevar <hinkocevar@gmail.com>
>> Sent: Tuesday, November 14, 2017 2:57 AM
>> To: Mark Rivers
>> Cc: tech-talk@aps.anl.gov
>> Subject: Re: areaDetector and HDF5 file saving segafaults
>>
>> Hi Mark,
>>
>> thanks for your time.
>> I have not yet had the time to reproduce this with SimDetector - on my
>> TODO list!
>>
>>
>> Here is the gdb with some variables printed out. I can't see any null
>> pointers or similar low hanging fruit.
>>
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> [Switching to Thread 0x7ffef4b68700 (LWP 26708)]
>> 0x00007ffff35d7e08 in __memcpy_ssse3_back () from /lib64/libc.so.6
>> Missing separate debuginfos, use: debuginfo-install
>> bzip2-libs-1.0.6-13.el7.x86_64 elfutils-libelf-0.168-8.el7.x86_64
>> elfutils-libs-0.168-8.el7.x86_64 glib2-2.50.3-3.el7.x86_64
>> glibc-2.17-196.el7.x86_64 glibmm24-2.50.0-1.el7.x86_64
>> libattr-2.4.46-12.el7.x86_64 libblkid-2.23.2-43.el7.x86_64
>> libcap-2.22-9.el7.x86_64 libffi-3.0.13-18.el7.x86_64
>> libgcc-4.8.5-16.el7.x86_64 libmount-2.23.2-43.el7.x86_64
>> libselinux-2.5-11.el7.x86_64 libsigc++20-2.10.0-1.el7.x86_64
>> libstdc++-4.8.5-16.el7.x86_64 libusbx-1.0.20-1.el7.x86_64
>> libuuid-2.23.2-43.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64
>> ncurses-libs-5.9-14.20130511.el7_4.x86_64 pcre-8.32-17.el7.x86_64
>> readline-6.2-10.el7.x86_64 systemd-libs-219-42.el7_4.1.x86_64
>> xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
>> (gdb) bt
>> #0  0x00007ffff35d7e08 in __memcpy_ssse3_back () at /lib64/libc.so.6
>> #1  0x00007ffff6371211 in H5Pset_fill_value (plist_id=<optimized out>,
>> type_id=216172782118131140, value=0x7ffcf7feffc0) at ../H5Pdcpl.c:3217
>> #2  0x00007ffff776bff7 in
>> NDFileHDF5AttributeDataset::createHDF5Dataset()
>> (this=this@entry=0x7ffcf7da1fe0) at
>> ../NDFileHDF5AttributeDataset.cpp:99
>> #3  0x00007ffff776c648 in
>> NDFileHDF5AttributeDataset::createDataset(int)
>> (this=this@entry=0x7ffcf7da1fe0, user_chunking=<optimized out>) at
>> ../NDFileHDF5AttributeDataset.cpp:75
>> #4  0x00007ffff776529e in NDFileHDF5::createAttributeDataset(NDArray*)
>> (this=this@entry=0xca2500, pArray=pArray@entry=0x7fff580aed70) at
>> ../NDFileHDF5.cpp:2535
>> #5  0x00007ffff776717f in NDFileHDF5::openFile(char const*, int,
>> NDArray*) (this=0xca2500, fileName=0x7ffef4b675d0
>> "/tmp/spectra1_14109.h5", openMode=<optimized out>,
>> pArray=0x7fff580aed70)
>>     at ../NDFileHDF5.cpp:268
>> #6  0x00007ffff775820d in NDPluginFile::openFileBase(int, NDArray*)
>> (this=this@entry=0xca2500, openMode=openMode@entry=2,
>> pArray=pArray@entry=0x7fff580aed70) at ../NDPluginFile.cpp:73
>> #7  0x00007ffff7758ee4 in NDPluginFile::writeFileBase()
>> (this=this@entry=0xca2500) at ../NDPluginFile.cpp:230
>> #8  0x00007ffff77597f8 in NDPluginFile::processCallbacks(NDArray*)
>> (this=0xca2500, pArray=0x7fff580aed70) at ../NDPluginFile.cpp:751
>> #9  0x00007ffff770871e in NDPluginDriver::processTask()
>> (this=0xca2500) at ../NDPluginDriver.cpp:467
>> #10 0x00007ffff48d44c9 in epicsThreadCallEntryPoint(void*)
>> (pPvt=0x7fff80000dc0) at ../../../src/libCom/osi/epicsThread.cpp:85
>> #11 0x00007ffff48d9f6c in start_routine (arg=0x7fff80000cd0) at
>> ../../../src/libCom/osi/os/posix/osdThread.c:403
>> #12 0x00007ffff2f66e25 in start_thread () at /lib64/libpthread.so.0
>> #13 0x00007ffff358534d in clone () at /lib64/libc.so.6
>>
>>
>> (gdb) frame 1
>> #1  0x00007ffff6371211 in H5Pset_fill_value (plist_id=<optimized out>,
>> type_id=216172782118131140, value=0x7ffcf7feffc0) at ../H5Pdcpl.c:3217
>> 3217         HDmemcpy(fill.buf, value, (size_t)fill.size);
>> (gdb) p fill
>> $1 = {sh_loc = {type = 0, file = 0x0, msg_type_id = 0, u = {loc =
>> {index = 0, oh_addr = 18446744073709551615}, heap_id = {id =
>> "\000\000\000\000\000\000\000", val = 0}}}, version = 2, type =
>> 0x7ffcec682ee0,
>>   size = 256, buf = 0x7ffcf7f13da0, alloc_time = H5D_ALLOC_TIME_LATE,
>> fill_time = H5D_FILL_TIME_IFSET, fill_defined = false}
>> (gdb) p fill.buf
>> $2 = (void *) 0x7ffcf7f13da0
>> (gdb) p value
>> $3 = (const void *) 0x7ffcf7feffc0
>> (gdb) p fill.size
>> $8 = 256
>> (gdb) x/256 fill.buf
>> 0x7ffcf7f13da0: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13da8: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13db0: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13db8: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13dc0: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13dc8: 0x11 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13dd0: 0x10 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13dd8: 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13de0: 0x00 0xd5 0x6a 0xec 0xfc 0x7f 0x00 0x00
>> 0x7ffcf7f13de8: 0x18 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13df0: 0x01 0xff 0xff 0xff 0xff 0xff 0xff 0xff
>> 0x7ffcf7f13df8: 0x44 0x45 0x41 0x44 0x42 0x45 0x45 0x46
>> 0x7ffcf7f13e00: 0x21 0x52 0x42 0x00 0x00 0x00 0x00 0x03
>> 0x7ffcf7f13e08: 0x01 0x00 0x00 0x00 0x01 0x00 0x00 0x00
>> 0x7ffcf7f13e10: 0x80 0x11 0x6d 0xec 0xfc 0x7f 0x00 0x00
>> 0x7ffcf7f13e18: 0x42 0x45 0x45 0x46 0x44 0x45 0x41 0x44
>> 0x7ffcf7f13e20: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13e28: 0xa5 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13e30: 0x48 0x35 0x4d 0x4d 0xff 0x7f 0x00 0x00
>> 0x7ffcf7f13e38: 0xa0 0x3a 0x6b 0xec 0xfc 0x7f 0x00 0x00
>> 0x7ffcf7f13e40: 0xf0 0xc6 0xbe 0xf7 0xfc 0x7f 0x00 0x00
>> 0x7ffcf7f13e48: 0x60 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13e50: 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13e58: 0x44 0x45 0x41 0x44 0x42 0x45 0x45 0x46
>> 0x7ffcf7f13e60: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13e68: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13e70: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13e78: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13e80: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13e88: 0x20 0xc7 0xbe 0xf7 0xfc 0x7f 0x00 0x00
>> 0x7ffcf7f13e90: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7f13e98: 0xff 0xff 0xff 0xff 0xff 0xff 0xff 0xff
>> (gdb) x/256 value
>> 0x7ffcf7feffc0: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7feffc8: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7feffd0: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7feffd8: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7feffe0: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7feffe8: 0x11 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7fefff0: 0x10 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7fefff8: 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0000: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0008: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0010: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0018: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0020: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0028: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0030: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0038: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0040: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0048: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0050: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0058: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0060: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0068: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0070: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0078: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0080: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0088: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0090: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0098: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff00a0: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff00a8: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff00b0: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff00b8: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>>
>>
>> (gdb) frame 2
>> #2  0x00007ffff776bff7 in
>> NDFileHDF5AttributeDataset::createHDF5Dataset
>> (this=this@entry=0x7ffcf7da1fe0) at
>> ../NDFileHDF5AttributeDataset.cpp:99
>> 99   H5Pset_fill_value(cparm_, datatype_, ptrFillValue_);
>> (gdb) p cparm_
>> $4 = 720575940382417266
>> (gdb) p datatype_
>> $5 = 216172782118131140
>> (gdb) p ptrFillValue_
>> $6 = (void *) 0x7ffcf7feffc0
>> (gdb) x/256 ptrFillValue_
>> 0x7ffcf7feffc0: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7feffc8: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7feffd0: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7feffd8: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7feffe0: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7feffe8: 0x11 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7fefff0: 0x10 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7fefff8: 0x01 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0000: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0008: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0010: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0018: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0020: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0028: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0030: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0038: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0040: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0048: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0050: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0058: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0060: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0068: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0070: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0078: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0080: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0088: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0090: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff0098: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff00a0: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff00a8: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff00b0: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> 0x7ffcf7ff00b8: 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
>> (gdb)
>>
>>
>> Hope this helps,
>> Hinko
>>
>> On Mon, Nov 13, 2017 at 1:46 PM, Mark Rivers <rivers@cars.uchicago.edu> wrote:
>>> Hi Hinko,
>>>
>>>
>>> Since you are using busy 1-6-1 I don't think the crash is due to the busy record.  It does not have the bug which was causing crashes for me.
>>>
>>>
>>> This is the code where it is crashing in H5Pdcpl.c
>>>
>>>         /* Set the fill value */
>>>         if(NULL == (fill.type = H5T_copy(type, H5T_COPY_TRANSIENT)))
>>>             HGOTO_ERROR(H5E_PLIST, H5E_CANTCOPY, FAIL, "can't copy datatype")
>>>         fill.size = (ssize_t)H5T_get_size(type);
>>>         if(NULL == (fill.buf = H5MM_malloc((size_t)fill.size)))
>>>             HGOTO_ERROR(H5E_RESOURCE, H5E_CANTINIT, FAIL, "memory allocation failed for fill value")
>>>         HDmemcpy(fill.buf, value, (size_t)fill.size);   CRASHES HERE
>>>
>>> That code is being called from here in NDFileHDF5AttributeDataset::createHDF5Dataset():
>>>
>>>
>>> asynStatus NDFileHDF5AttributeDataset::createHDF5Dataset()
>>> {
>>>   asynStatus status = asynSuccess;
>>>
>>>   cparm_ = H5Pcreate(H5P_DATASET_CREATE);
>>>
>>>   H5Pset_fill_value(cparm_, datatype_, ptrFillValue_);  CRASHES IN CALL FROM HERE
>>>
>>>
>>>
>>> This is the private data in the NDFileHDF5AttributeDataset class:
>>>
>>> private:
>>>   asynStatus createHDF5Dataset();
>>>   asynStatus configureDims(int user_chunking);
>>>   asynStatus configureDimsFromDataset(bool multiframe, int extradimensions, int *extra_dims, int *user_chunking);
>>>   asynStatus typeAsHdf();
>>>   void extendDataSet();
>>>   void extendDataSet(hsize_t *offsets);
>>>   void extendIndexDataSet(hsize_t offset);
>>>
>>>   std::string      name_;            // Name of the attribute
>>>   std::string      dsetName_;        // Name of the dataset to store
>>>   hid_t            file_;            // File handle
>>>   NDAttrDataType_t type_;            // NDAttribute type
>>>   std::string      groupName_;       // Name of the parent group
>>>   hid_t            dataset_;         // Dataset handle
>>>   hid_t            dataspace_;
>>>   hid_t            memspace_;
>>>   hid_t            datatype_;
>>>   hid_t            cparm_;
>>>   hid_t            filespace_;
>>>   void             *ptrFillValue_;
>>>   hsize_t          *dims_;
>>>   hsize_t          *offset_;
>>>   hsize_t          *chunk_;
>>>   hsize_t          *maxdims_;
>>>   hsize_t          *virtualdims_;
>>>   hsize_t          *elementSize_;
>>>   bool             isUndefined_;
>>>   int              rank_;            // number of dimensions
>>>   int              nextRecord_;
>>>   int              extraDimensions_;
>>>   hdf5::When_t     whenToSave_;
>>>
>>> };
>>>
>>> I have 2 suggestions to track down the problem:
>>>
>>> - Use gdb to examine the values of the variables in H5Pdcpl.c and the NDFileHDF5AttributeDataset private member data when it crashes
>>>
>>> or
>>>
>>> - Use asynPrint to print the values every time that function is called, and route the asynTrace output to a file, since it may be large.
>>>
>>>
>>> Have you tried to reproduce it with the simDetector?
>>>
>>>
>>> Mark
>>>
>>>
>>>
>>>
>>>
>>> ________________________________
>>> From: Hinko Kocevar <hinkocevar@gmail.com>
>>> Sent: Monday, November 13, 2017 2:32 AM
>>> To: Mark Rivers
>>> Cc: tech-talk@aps.anl.gov
>>> Subject: Re: areaDetector and HDF5 file saving segafaults
>>>
>>> Hi Mark,
>>>
>>> On Thu, Nov 9, 2017 at 1:06 PM, Mark Rivers <rivers@cars.uchicago.edu> wrote:
>>>> Hi Hinko,
>>>>
>>>>> I'm using Single saving mode, saving one image per acquisition at ~10 images per second.
>>>>
>>>> How is the file saving being done?  Are you setting the AutoSave PV to 1?   If so, what version of the "busy" module are you using?  I have recently seen a segfault when using AutoSave at much higher frame rates with the TIFF file plugin, and the same problem might well occur with the HDF5 plugin. I understand the problem, it has to do with the busy record asyn device support not correctly handling callbacks when they occur rapidly.  However, I don't think this happens with any official release of the busy module, only with changes made on the master branch. I am actively working on a fix for that problem.
>>>
>>> Yes, I'm using autosave=1. Busy version is 1-6-1.
>>>
>>>>
>>>> It seems unlikely that hardware problems would cause the error with the stack trace you sent, but it's always possible I suppose.
>>>>
>>>> There was also a change in ADCore R3-0 to fix a problem storing NDAttributes in HDF5 files when those attributes were coming from EPICS PVs that were disconnected.  You are using that fixed version.  Is it possible that one of your attributes is coming from a disconnected EPICS PV?  If so, perhaps I introduced a bug.
>>>
>>> I think all PVs should be connected - I'm only accessing PVs local to the IOC.
>>>
>>> Thanks,
>>> Hinko
>>>
>>>>
>>>> Mark
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> From: Hinko Kocevar <hinkocevar@gmail.com>
>>>> Sent: Thursday, November 9, 2017 1:52 AM
>>>> To: Mark Rivers
>>>> Cc: tech-talk@aps.anl.gov
>>>> Subject: Re: areaDetector and HDF5 file saving segafaults
>>>>
>>>> Hi Mark,
>>>>
>>>> Image size is 1936 x 1216, data type is uint16 (~2.3MB). Camera is AlliedVision Manta G-235B, controlled with aravis+aravisGigE.
>>>> I'm saving some attributes - see the attached XML file.
>>>> I'm using Single saving mode, saving one image per acquisition at ~10 images per second.
>>>> It happens quite often, if I leave if running for couple of minutes I get the segfault.
>>>> I have not yet tried reproducing with simDetector - will do.
>>>> I'm using HDF5 ver 1.10.1 that comes with ADSupport.
>>>>
>>>> The thing is that I have not noticed this problems before, this now happens on a remote machine at an experiment we are conducting. It could be the hardware..
>>>>
>>>> Thank you,
>>>> Hinko
>>>>
>>>> On Tue, Nov 7, 2017 at 2:59 PM, Mark Rivers <rivers@cars.uchicago.edu<mailto:rivers@cars.uchicago.edu>> wrote:
>>>> Hi Hinko,
>>>>
>>>>
>>>> Also, what version of HDF5 are you using, the one from ADSupport (and if so what version) or a system library version?
>>>>
>>>>
>>>> Mark
>>>>
>>>>
>>>>
>>>> ________________________________
>>>> From: tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov> <tech-talk-bounces@aps.anl.gov<mailto:tech-talk-bounces@aps.anl.gov>> on behalf of Hinko Kocevar <hinkocevar@gmail.com<mailto:hinkocevar@gmail.com>>
>>>> Sent: Tuesday, November 7, 2017 3:58 AM
>>>> To: tech-talk@aps.anl.gov<mailto:tech-talk@aps.anl.gov>
>>>> Subject: areaDetector and HDF5 file saving segafaults
>>>>
>>>> Hi,
>>>>
>>>> I'm experiencing segmentation faults that trace to HDF5 file saving plugin. This is with ADAravis camera, 10 images per second.
>>>>
>>>> EPICS base is 3.15.4, AreaDetector is 3-1.
>>>>
>>>> epics>
>>>> Program received signal SIGSEGV, Segmentation fault.
>>>> [Switching to Thread 0x7fffed55a700 (LWP 27283)]
>>>> 0x00007ffff35d7d89 in __memcpy_ssse3_back () from /lib64/libc.so.6
>>>> (gdb) bt
>>>> #0  0x00007ffff35d7d89 in __memcpy_ssse3_back () at /lib64/libc.so.6
>>>> #1  0x00007ffff6371211 in H5Pset_fill_value (plist_id=<optimized out>, type_id=216172782117023038, value=0x7ffd67ff2fb0) at ../H5Pdcpl.c:3217
>>>> #2  0x00007ffff776bff7 in NDFileHDF5AttributeDataset::createHDF5Dataset() (this=this@entry=0x7ffd5c007460) at ../NDFileHDF5AttributeDataset.cpp:99
>>>> #3  0x00007ffff776c648 in NDFileHDF5AttributeDataset::createDataset(int) (this=this@entry=0x7ffd5c007460, user_chunking=<optimized out>)
>>>>     at ../NDFileHDF5AttributeDataset.cpp:75
>>>> #4  0x00007ffff776529e in NDFileHDF5::createAttributeDataset(NDArray*) (this=this@entry=0x939530, pArray=pArray@entry=0x7fff10968800)
>>>>     at ../NDFileHDF5.cpp:2535
>>>> #5  0x00007ffff776717f in NDFileHDF5::openFile(char const*, int, NDArray*) (this=0x939530, fileName=0x7fffed5595d0 "/home/dev/DTUdata/image1_20896.h5", openMode=<optimized out>, pArray=0x7fff10968800) at ../NDFileHDF5.cpp:268
>>>> #6  0x00007ffff775820d in NDPluginFile::openFileBase(int, NDArray*) (this=this@entry=0x939530, openMode=openMode@entry=2, pArray=pArray@entry=0x7fff10968800) at ../NDPluginFile.cpp:73
>>>> #7  0x00007ffff7758ee4 in NDPluginFile::writeFileBase() (this=this@entry=0x939530) at ../NDPluginFile.cpp:230
>>>> #8  0x00007ffff77597f8 in NDPluginFile::processCallbacks(NDArray*) (this=0x939530, pArray=0x7fff10968800) at ../NDPluginFile.cpp:751
>>>> #9  0x00007ffff770871e in NDPluginDriver::processTask() (this=0x939530) at ../NDPluginDriver.cpp:467
>>>> #10 0x00007ffff48d44c9 in epicsThreadCallEntryPoint(void*) (pPvt=0x7fffd4001400) at ../../../src/libCom/osi/epicsThread.cpp:85
>>>> #11 0x00007ffff48d9f6c in start_routine (arg=0x7fffd4001310) at ../../../src/libCom/osi/os/posix/osdThread.c:403
>>>> #12 0x00007ffff2f66e25 in start_thread () at /lib64/libpthread.so.0
>>>> #13 0x00007ffff358534d in clone () at /lib64/libc.so.6
>>>>
>>>> Thanks,
>>>> Hinko
>>>>
>>>> --
>>>> .. the more I see the less I believe.., AE AoR
>>>>
>>>>
>>>>
>>>> --
>>>> .. the more I see the less I believe.., AE AoR
>>>
>>>
>>>
>>> --
>>> .. the more I see the less I believe.., AE AoR
>>
>>
>>
>> --
>> .. the more I see the less I believe.., AE AoR
>
>
>
> --
> .. the more I see the less I believe.., AE AoR



-- 
.. the more I see the less I believe.., AE AoR

Replies:
RE: areaDetector and HDF5 file saving segafaults Mark Rivers
References:
areaDetector and HDF5 file saving segafaults Hinko Kocevar
Re: areaDetector and HDF5 file saving segafaults Mark Rivers
Re: areaDetector and HDF5 file saving segafaults Hinko Kocevar
Re: areaDetector and HDF5 file saving segafaults Mark Rivers
Re: areaDetector and HDF5 file saving segafaults Hinko Kocevar
Re: areaDetector and HDF5 file saving segafaults Mark Rivers
Re: areaDetector and HDF5 file saving segafaults Hinko Kocevar
Re: areaDetector and HDF5 file saving segafaults Mark Rivers
Re: areaDetector and HDF5 file saving segafaults Hinko Kocevar

Navigate by Date:
Prev: Re: areaDetector and HDF5 file saving segafaults Hinko Kocevar
Next: RE: areaDetector and HDF5 file saving segafaults Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  <2017
Navigate by Thread:
Prev: Re: areaDetector and HDF5 file saving segafaults Hinko Kocevar
Next: RE: areaDetector and HDF5 file saving segafaults Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  <2017
ANJ, 17 Nov 2017 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·