Managing Heterogeneous Scientific Data at the Advanced Photon Source, The USA’s Busiest Synchrotron User Facility, Nicholas Schwarz, Sinisa Veseli, Barbara Frosik, Arthur Glowacki, John Hammonds, Dariusz Jarosz, Faisal Khan, Ke Yue

Venue: Poster at the 9th Research Data Alliance Plenary Meeting

Advances in High-Performance Data Analysis and Data Management at the APS, Nicholas Schwarz

Venue: Proceedings of Proceedings of New Opportunities for Better User Group Software (NOBUGS) 2016

High-Performance XPCS Data Reduction Using Virtualized Computing Resources, Faisal Khan, Nicholas Schwarz, Suresh Narayanan, Alec Sandy, Collin Schmitz, Benjamin Pausma, Ryan Aydelott.

Venue: Proceedings of New Opportunities for Better User Group Software (NOBUGS) 2016

RSMap3D: Reciprocal Space Mapping Software, John Hammonds, Jonathan Tischler, Nicholas Schwarz

Venue: Poster in the Proceedings of New Opportunities for Better User Group Software (NOBUGS) 2016

Advances in Data Analysis and Management for Materials Exploration at the APS, Nicholas Schwarz

Venue: Presentation at the 2016 Advanced Light Source User Meeting

APS Scientific Computing and Data Processing Strategies, Nicholas Schwarz

Venue: Presentation at the 16th Three-way meeting

Recent Advancements and Deployments of EPICS Version 4, G.R. White, M.V. Shankar (SLAC, Menlo Park, California, USA), A. Arkilic, L.R. Dalesio, M.A. Davidsaver, M.R. Kraimer, N. Malitsky, B.S. Martins (BNL, Upton, Long Island, New York, USA), S.M. Hartman, K.-U. Kasemir (ORNL, Oak Ridge, Tennessee, USA), D.G. Hickin (DLS, Oxfordshire, United Kingdom), A.N. Johnson, S. Veseli (ANL, Argonne, Ilinois, USA), T. Korhonen (ESS, Lund, Sweden), R. Lange (ITER Organization, St. Paul lez Durance, France), M. Sekoranja (Cosylab, Ljubljana, Slovenia), G. Shen (FRIB, East Lansing, Michigan, USA)

Venue: ICALEPCS 2015

PvaPy: Python API for EPICS PV Access, Sinisa Veseli

Venue: ICALEPCS 2015

Component Database for APS Upgrade, S. Veseli, N.D. Arnold, J. Carwardine, G. Decker, D.P. Jarosz, N. Schwarz

Venue: ICALEPCS 2015

Accelerating Laue Depth Reconstruction Algorithm With CUDA, Ke Yue, Nicholas Schwarz, Jonathan Tischler

Citation: Proceedings of IEEE High Performance Extreme Computing Conference (HPEC ‘15), Waltham, MA, 9/15/15 - 9/17/15

Accelerating Laue Depth Reconstruction Algorithm With CUDA, Ke Yue, Nicholas Schwarz, Jonathan Tischler

Citation: Accelerating Laue Depth Reconstruction Algorithm With CUDA, Proceedings of IEEE Cluster 2015, Chicago, IL, 9/8/15 – 9/11/15

High-Performance Computing Software to Improve Beamline and Accelerator Efficiency, Nicholas Schwarz, Ned Arnold, Arthur Glowacki, Sinisa Veseli, Ke Yue

Citation: APS News 2014

Practices and Standards for Data and Processing at the APS, Brian Toby, Doga Gursoy, Francesco De Carlo, Nicholas Schwarz, Hemant Sharma, Chris Jacobsen

The handling of datasets at scientific user facilities is becoming progressively more challenging as advances in sources and detectors drive increasingly aggressive data acquisition rates. The ability to share and process these data depends on the ability of the originator of the data and the persons making use of that data to unequivocally agree on what these data mean. This understanding is driven by use of standardized data formats.

Citation: Synchrotron Radiation News, 28:2, pp. 15 – 21 (2015)

Scientific Data Analysis & Management at the APS, Nicholas Schwarz (AES-SSG)

Venue: NSLS-II & CFN Joint Users' Meeting - Workshop 6: From First Photon to Publication: Tackling the Synchrotron Data Challenge

EPICS Training

The Experimental Physics and Industrial Control System (EPICS) is a set of Open Source software tools, libraries and applications developed collaboratively and used worldwide. EPICS is used to create distributed soft real-time control systems for scientific instruments such as a particle accelerators, telescopes and other large scientific experiments.

EPICS is used extensively at the APS. The last comprehensive EPICS training at the APS was held ten years ago. Since then many things have changed, and many individuals have joined the organization. The AES Software Services Group is organizing an updated series of EPICS training classes to begin this September and continue through the winter of 2015. Many of the classes will have corresponding hands-on laboratory sessions.

Click here to to view the full schedule of classes.

Coordinates from light microscope to beamline: The uProbeX software, Arthur Glowacki (AES-SSG)

Venue: InterCAT Technical Workgroup (TWG)

Click here to download the presentation.

New tools for fast and reliable 3-dimensional reciprocal space mapping using area detectors, Christian Schlepütz (XSD-SSM) and John Hammonds (AES-SSG)

Venue: InterCAT Technical Workgroup (TWG)

Click here to download the presentation.

EPICS Version 4 Development, Andrew Johnson (AES-SSG)

EPICS Version 4 has been a formal international development project since 2011, with active members from 7 organizations spread across 4 US states and 3 European countries. This talk will discuss the main objectives and current state of the project, explain how and why the APS is involved, and suggest some advantages it could bring to the APS Accelerator and Beamline control systems. This is an encore presentation of the talk given on 26 November 2013.

EPICS Version 4 Development, Andrew Johnson (AES-SSG)

EPICS Version 4 has been a formal international development project since 2011, with active members from 7 organizations spread across 4 US states and 3 European countries. This talk will discuss the main objectives and current state of the project, explain how and why the APS is involved, and suggest some advantages it could bring to the APS Accelerator and Beamline control systems.

Click here to download the presentation.

Globus Tools for Transferring Data at the APS, Faisal Khan (AES-SSG)

Click here to download the presentation.

Experiment Control and Analysis for High-Resolution Tomography, Nicholas Schwarz (AES-SSG), Faisal Khan (AES-SSG), Ke Yue (AES-SSG), John Hammonds (AES-SSG), Francesco De Carlo (XSD-IMG)

X-ray Computed Tomography (XCT) is a powerful technique for imaging 3D structures at the micro- and nano-levels. Recent upgrades to tomography beamlines at the APS have enabled imaging at resolutions up to 20nm at increased pixel counts and speeds. As detector resolution and speed increase, the amount of data that must be transferred and analyzed also increases. This coupled with growing experiment complexity drives the need for software to automate data acquisition and processing. We present an experiment control and data processing system for tomography beamlines that helps address this concern. The software, written in C++ using Qt, interfaces with EPICS for beamline control and provides live and offline data viewing, basic image manipulation features, and scan sequencing that coordinates EPICS enabled apparatus. Post acquisition, the software triggers a workflow pipeline, written using ActiveMQ, that transfers data from the detector computer to an analysis computer, and launches a reconstruction process. Experiment metadata and provenance information is stored along with raw and analyzed data in a single HDF5 file.

Venue: ICALEPCS 2013

Click here for the official conference site.

Effective End-to-end Management of Data Acquisition and Analysis for X-ray Photon Correlation Spectroscopy, Faisal Khan (AES-SSG), John Hammonds (AES-SSG), Suresh Narayanan (XSD-TRR), Alec Sandy (XSD-TRR), Nicholas Schwarz (AES-SSG)

Low latency between data acquisition and analysis is of critical importance to any experiment. The combination of a faster parallel algorithm and a data pipeline for connecting disparate components (detectors, clusters, file formats) enabled us to greatly enhance the operational efficiency of the x-ray photon correlation spectroscopy experiment facility at the Advanced Photon Source. The improved workflow starts with raw data (120 MB/s) streaming directly from the detector camera, through an on-the-fly discriminator implemented in firmware to Hadoop’s distributed file system in a structured HDF5 data format. The user then triggers the MapReduce-based parallel analysis. For effective bookkeeping and data management, the provenance information and reduced results are added to the original HDF5 file. Finally, the data pipeline triggers user-specific software for visualizing the data. The whole process is completed shortly after data acquisition – a significant improvement of operation over the previous setup. The faster turnaround time helps scientists to make near real-time adjustments to the experiments.

Venue: ICALEPCS 2013

Click here for the official conference site.

Not Dead Yet: Recent Enhancements and Future Plans for EPICS Version 3, Andrew Johnson (AES-SSG), Janet Anderson (AES-SSG), Michael Davidsaver (BNL), Ralph Lange (BESSY)

The EPICS Version 4 development effort* is not planning to replace the current Version 3 IOC Database or its use of the Channel Access network protocol in the near future. Interoperability is a key aim of the V4 development, which is building upon the older IOC implementation. EPICS V3 continues to gain new features and functionality on its Version 3.15 development branch, while the Version 3.14 stable branch has been accumulating minor tweaks, bug fixes, and support for new and updated operating systems. This paper describes the main enhancements provided by recent and upcoming releases of EPICS Version 3 for control system applications.

Venue: ICALEPCS 2013

Click here for the official conference site.

Using Message Broker with EPICS: SPX Controls Use Cases, Siniša Veseli (AES-SSG)

Venue: Fall 2013 EPICS Collaboration Meeting - ICALEPCS 2013 Satellite Meeting

Click here for the official conference site.

Data Management for the Tao of Fusion Project at the APS, Faisal Khan (AES-SSG)

Data Analysis and Management at the APS: Successes and Challenges, Nicholas Schwarz (AES-SSG)

Venue: NSF SIXNS Workshop - Advanced Analysis of X-Ray and Neutron Scattering Data: Getting from Data to Science

• Click here to download the presentation.
• Click here for the official meeting site.

Highlights from the 2013 Spring EPICS Collaboration Meeting, Andrew Johnson (AES-SSG)

The EPICS Collaboration Meeting this May was hosted jointly by the Diamond Light Source and the ISIS Spallation Neutron Source, and was held at their Harwell campus in Oxfordshire, England. Andrew Johnson will present the highlights from a selection of the talks given at that meeting, including an update on the current state of the EPICS V4 developments.

Data Management and Automation at the APS, Nicholas Schwarz (AES-SSG)

Venue: 2013 Three-way Meeting

• Click here to download the presentation.
• Click here for the official meeting site.

Driving Discovery: Visualization, Data Management, and Workflow Techniques, Nicholas Schwarz (APS), Siwei Wang (MCS), Ian McNulty (CNM), and C.D. Phatak (EMC)

The great richness of data collected at the APS, CNM, and EMC plays critical roles in scientific exploration. As an example, imaging and microscopy experiments are adding dynamics and spectroscopic information to tomography. However, methods for understanding data have not kept pace; there is no “Moore’s law” scaling that applies to by-hand examination of data. Manual management and analysis of data is too time consuming and cumbersome for large, complex datasets. State-of-the-art mathematics and computer science tools will help automate the understanding process for large datasets. Only then will scientific understanding be able to fully benefit from the coming deluge of data. This workshop is organized to discuss the state of the art and future potential of advanced optimization, visualization, data management, and workflow techniques for Argonne’s user facilities. It brings together experts in optimization, computer vision, visualization and data management alongside user scientists to discuss current and future applications. The goal is to elaborate on how users can benefit from these techniques to enable new scientific discovery.

For the official workshop site, links to the talks, and the workshop report.