Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  <2017 Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  <2017
<== Date ==> <== Thread ==>

Subject: RE: Area Detector and high performance NVME devices
From: Mark Rivers <rivers@cars.uchicago.edu>
To: "Engbretson, Mark S." <engbretson@aps.anl.gov>
Cc: "tech-talk@aps.anl.gov" <tech-talk@aps.anl.gov>
Date: Fri, 30 Jun 2017 12:44:07 +0000
Hi Mark,

> What i want is an effect more like just opening 2 or more distinct files at once. If they are doing totally different things, 
> with totally unrelated life cycles, what is the mutex protecting?

In general the mutex is protecting global structures and variables in the HDF5 library.  Consider an extremely simple example pseudocode:

FILE *globalFileId;
mutex globalLock;
void writeChunk(FILE *fileId, void *pData)
{
  lock(globalLock);
  globalFileId = fileId;
  writeData(pData);
  unlock(globalLock);
}
static void writeData(void *pData)
{
  fwrite(pData, 1, 1000, globalFileId);
}

writeChunk is externally visible and called by user-code, while writeData is private and hidden.  Because writeChunk does not pass fileId directly to writeData, but rather uses a global variable, it needs to be protected by a mutex.  Otherwise this code would not be thread safe.

Obviously the HDF5 library does not do anything this silly, but the principle is the same.  It uses global variables and structures that need to be protected by a mutex.  It does this in such a way that only 1 user-callable HDF5 function can be running at once, i.e. there is no concurrency possible in the library.   The only way to achieve concurrency is to run multiple processes, not multiple threads.  The HDF5 documents are very clear about this, and the performance results I posted yesterday using multiple threads vs multiple processes confirm it.

Mark

________________________________________
From: Engbretson, Mark S. [engbretson@aps.anl.gov]
Sent: Thursday, June 29, 2017 7:08 PM
To: Mark Rivers
Cc: ulrik.pedersen@diamond.ac.uk; tech-talk@aps.anl.gov
Subject: Re: Area Detector and high performance NVME devices

But wait . . . It can not work that way. Threads pop in and out of existence all the time. They may or may not use the HDF5 libraries, but if they do, they certainly would not want to have a dependency on code or constructs that could go "poof" at any instant.

What i want is an effect more like just opening 2 or more distinct files at once. If they are doing totally different things, with totally unrelated life cycles, what is the mutex protecting?

On Jun 29, 2017, at 18:16, Mark Rivers <rivers@cars.uchicago.edu> wrote:

What you propose would not work, because the problem is not in NDFileHDF5.cpp, it is in the underlying HDF5 library.  There are global structures there that you would not be making multiple instances of, so they need to be protected by the mutex.

-----Original Message-----
From: Mark S. Engbretson [mailto:Engbretson@anl.gov]
Sent: Thursday, June 29, 2017 5:05 PM
To: ulrik.pedersen@diamond.ac.uk; Mark Rivers
Cc: tech-talk@aps.anl.gov
Subject: RE: Area Detector and high performance NVME devices

For the sake of argument, what would happen if instead of creating multiple
instances of the HDF File plugin, which is actually the exact same entity, I
instead cloned the NDFileHDF5.cpp sources and made it a couple of unique
plugins named <NDFileHDF5>_A,B,C,and D. I can do this on windows system and
force every thread to have its own unique copy everything, so there is no
conflicts in non thread safe code just as you would not expect any mutex
issues from another process running on the system.

I know that I should technically be able to use constructs such as

void *handle1 = dlmopen(LM_ID_NEWLM, "/path/to/library.so", RTLD_NOW);
void *handle2 = dlmopen(LM_ID_NEWLM, "/path/to/library.so", RTLD_NOW);

Where the references is to the same shared library, but via difference
handles don't conflict.  But there should be easier ways to achieve the same
effect. I.e, with multiple file plugins, there are no common anythings that
should need that mutex lock or could be trashed.



-----Original Message-----
From: ulrik.pedersen@diamond.ac.uk [mailto:ulrik.pedersen@diamond.ac.uk]
Sent: Thursday, June 29, 2017 9:34 AM
To: Engbretson@anl.gov; rivers@cars.uchicago.edu
Cc: tech-talk@aps.anl.gov
Subject: RE: Area Detector and high performance NVME devices

References:
Area Detector and high performance NVME devices Mark S. Engbretson
RE: Area Detector and high performance NVME devices Mark Rivers
RE: Area Detector and high performance NVME devices Mark S. Engbretson
RE: Area Detector and high performance NVME devices Mark Rivers
RE: Area Detector and high performance NVME devices Mark S. Engbretson
Re: Area Detector and high performance NVME devices ulrik.pedersen
RE: Area Detector and high performance NVME devices Mark S. Engbretson
RE: Area Detector and high performance NVME devices ulrik.pedersen
RE: Area Detector and high performance NVME devices Mark Rivers
RE: Area Detector and high performance NVME devices Mark S. Engbretson
RE: Area Detector and high performance NVME devices ulrik.pedersen
RE: Area Detector and high performance NVME devices Mark S. Engbretson
RE: Area Detector and high performance NVME devices Mark Rivers
Re: Area Detector and high performance NVME devices Engbretson, Mark S.

Navigate by Date:
Prev: RE: Area Detector and high performance NVME devices ulrik.pedersen
Next: areaDetector ADViewers R1-0 released Mark Rivers
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  <2017
Navigate by Thread:
Prev: RE: Area Detector and high performance NVME devices ulrik.pedersen
Next: PVs disconnected in CSS BOY despite access through caget Iain Marcuson
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  2016  <2017
ANJ, 01 Jul 2017 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·