EPICS Controls Argonne National Laboratory

Experimental Physics and
Industrial Control System

1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024  Index 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024 
<== Date ==> <== Thread ==>

Subject: Re: Philosophy regarding use of open source libraries for EPICS
From: Bruce Hill <[email protected]>
To: <[email protected]>
Date: Wed, 16 Nov 2016 15:30:10 -0800
Hi Rod,

We've faced the same problem here at SLAC. Our environment has a wide variety of host machines, including ioc hosts, build hosts, operator consoles, and analysis stations. Originally these were RHEL5, most now are RHEL6, with RHEL7 being introduced. Each of these has a different libc, requiring different builds for each. Relying on yum repo installation of third party packages wasn't working as the available package versions were often way out of date.

Most of these hosts are diskless, booting a standard image, so to avoid needing to install all the third party packages needed for both EPICS development and analysis on each host, we build and install these to networked file systems.

To handle this, we created a python based system we call package manager. It consists of python scripts that know how to build and install typical c/c++ unix packages from tarballs using ./configure, make, make install, and also how to build and install python packages using setup.py build and setup.py install. For each package type, there's also a dependency file to specify version specific package dependencies, and a short python file to handle quirks of each package type. For an easy package the new python code is about 10 lines of boilerplate, but we've also handled builds which use qmake, autogen, cmake, etc. We can also customize configure arguments, build arguments, add missing headers, customize installs etc.

In practice it means for each new package type, an engineer will download the tarball, tweak the package specific python to get it to build, and check it in. Then we can build that package, and usually subsequent versions of the package, for different host targets and at multiple build sites for different areas at SLAC.

We can now build over 180 different third party packages, including the most recent V4 EPICS-CPP 4.6.0.

There's also a release feature, where a release file specifies a version specific set of packages. When the release is built, it builds all the required packages including their dependencies, then creates bin, lib, include, and site-packages directories w/ soft links to all the relevant files from each package and it's dependents. Env scripts add bin to PATH, lib to LD_LIBRARY_PATH and PYTHONPATH, site-packages to PYTHONPATH, etc.

This has been particularly useful for python applications, allowing us to create applications which launch w/ their own custom python environment, thus avoiding potential conflicts w/ other python packages.

For EPICS builds, we typically strip out and/or disable builds of vendor packages, modifying CONFIG_SITE to specify versions and locations for hdf5, jpeg, gif, aravis, ffmpeg, and other packages.

In regards to your question about how far down to go re dependent packages, we usually stop for common packages which are readily available in the standard Redhat images.

Regards,

- Bruce


On 11/16/2016 07:43 AM, Rod Nussbaumer wrote:
All:

I would like to get peoples' opinions about the right way to deal with open source third party libraries that get bound to EPICS applications, particularly on Linux, but wherever else it comes into play. An example case would be libgif, which can be used by EDM to display GIF images in EDM screens. The Area Detector package seems to make significant use of such libraries for image processing. So far, the standard procedure here has been to install the libraries from binary repositories supported by OS vendors such as Redhat, Debian, etc. We do this on development hosts and the matching production host architectures. However, this leaves us vulnerable to incompatible changes that may be introduced due to routine maintenance of the hosts, and silently changes the deployed EPICS software with no knowledge of the effect, or testing that it works as expected. The alternative would be to build from sources, all libraries that go into an EPICS application, and install the lib binaries in a way that they get used in lieue of the system-installed packages. Does anyone do that? Am I being paranoid? The obvious question is how far would you take this approach, since literally everything binds to libc, libm, and some other standard libraries.
Thanks in advance for your opinions.

Rod Nussbaumer
TRIUMF
Vancouver, Canada


--
Bruce Hill
Member Technical Staff
SLAC National Accelerator Lab
2575 Sand Hill Road M/S 10
Menlo Park, CA  94025


References:
Philosophy regarding use of open source libraries for EPICS Rod Nussbaumer

Navigate by Date:
Prev: Re: Philosophy regarding use of open source libraries for EPICS J. Lewis Muir
Next: Re: Philosophy regarding use of open source libraries for EPICS Konrad, Martin
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024 
Navigate by Thread:
Prev: Re: Philosophy regarding use of open source libraries for EPICS J. Lewis Muir
Next: Re: Philosophy regarding use of open source libraries for EPICS Konrad, Martin
Index: 1994  1995  1996  1997  1998  1999  2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  2010  2011  2012  2013  2014  2015  <20162017  2018  2019  2020  2021  2022  2023  2024 
ANJ, 16 Nov 2016 Valid HTML 4.01! · Home · News · About · Base · Modules · Extensions · Distributions · Download ·
· Search · EPICS V4 · IRMIS · Talk · Bugs · Documents · Links · Licensing ·