An innovative collaboration between computer and computational scientists from the Data Science and Learning Division (DSL) and Advanced Photon Source (APS) at Argonne, and the Electrical and Computer Engineering department at the University of Illinois at Urbana-Champaign (UIUC) has garnered the Best Paper Award at the SC20 virtual conference.
The research team — led by Tekin Bicer of DSL and Ph.D. candidate Mert Hidayetoglu (UIUC) and comprising Simon Garcia de Gonzalo (Barcelona Supercomputing Center), Bin Ren (College of William & Mary), Vincent De Andrade (APS/X-ray Science Division), Doga Gursoy (APS/X-ray Science Division), Rajkumar Kettimuthu (DSL), Ian Foster (DSL) and Wen-mei Hwu (UIUC) — is focused on using a high-performance, iterative reconstruction system for noninvasive imaging at synchrotron facilities. The new design, described in the team’s winning paper “Petascale XCT: 3D Image Reconstruction with Hierarchical Communications on Multi-GPU Nodes,” can be used for terabyte(s)-scale 3D volumes and is not constrained by computational requirements.
The work involves the following three novel optimizations: (1) extends the 2D memory-centric approach to 3D; (2) includes hierarchical communications by exploiting “fat-node” architecture with many GPUs; and (3) employs mixed-precision types while preserving convergence rate and quality.
The team extensively evaluated the proposed optimizations and scaling on the Summit supercomputer at Oak Ridge National Laboratory. Their largest reconstruction is a mouse brain volume with 9K×11K×11K voxels, where the total reconstruction time is under 3 minutes using 24,576 GPUs and reaching 65 PFLOPS, which is 34% of Summit’s peak performance.
The original Argonne press release can be read here.
The Advanced Photon Source is a U.S. DOE User Facility operated for the DOE Office of Science by Argonne National Laboratory under contract no. DE-AC02-06CH11357.
The U.S. Department of Energy's APS is one of the world’s most productive x-ray light source facilities. Each year, the APS provides high-brightness x-ray beams to a diverse community of more than 5,000 researchers in materials science, chemistry, condensed matter physics, the life and environmental sciences, and applied research. Researchers using the APS produce over 2,000 publications each year detailing impactful discoveries, and solve more vital biological protein structures than users of any other x-ray light source research facility. APS x-rays are ideally suited for explorations of materials and biological structures; elemental distribution; chemical, magnetic, electronic states; and a wide range of technologically important engineering systems from batteries to fuel injector sprays, all of which are the foundations of our nation’s economic, technological, and physical well-being.
Argonne National Laboratory seeks solutions to pressing national problems in science and technology. The nation's first national laboratory, Argonne conducts leading-edge basic and applied scientific research in virtually every scientific discipline. Argonne researchers work closely with researchers from hundreds of companies, universities, and federal, state and municipal agencies to help them solve their specific problems, advance America's scientific leadership and prepare the nation for a better future. With employees from more than 60 nations, Argonne is managed by UChicago Argonne, LLC, for the U.S. DOE Office of Science.
The U.S. Department of Energy's Office of Science is the single largest supporter of basic research in the physical sciences in the United States and is working to address some of the most pressing challenges of our time. For more information, visit the Office of Science website.