Award Abstract # 1931561
Collaborative Research: Frameworks: Machine learning and FPGA computing for real-time applications in big-data physics experiments

NSF Org: OAC
Office of Advanced Cyberinfrastructure (OAC)
Recipient: UNIVERSITY OF ILLINOIS
Initial Amendment Date: September 17, 2019
Latest Amendment Date: January 7, 2022
Award Number: 1931561
Award Instrument: Standard Grant
Program Manager: Varun Chandola
vchandol@nsf.gov
 (703)292-2656
OAC
 Office of Advanced Cyberinfrastructure (OAC)
CSE
 Directorate for Computer and Information Science and Engineering
Start Date: October 1, 2019
End Date: September 30, 2023 (Estimated)
Total Intended Award Amount: $651,314.00
Total Awarded Amount to Date: $651,314.00
Funds Obligated to Date: FY 2019 = $651,314.00
History of Investigator:
  • Volodymyr Kindratenko (Principal Investigator)
    kindrtnk@illinois.edu
  • Daniel Katz (Co-Principal Investigator)
  • Eliu Huerta Escudero (Co-Principal Investigator)
  • Eliu Huerta Escudero (Former Principal Investigator)
  • Volodymyr Kindratenko (Former Co-Principal Investigator)
Recipient Sponsored Research Office: University of Illinois at Urbana-Champaign
506 S WRIGHT ST
URBANA
IL  US  61801-3620
(217)333-2187
Sponsor Congressional District: 13
Primary Place of Performance: Board of Trustees of the University of Illinois
506 S Wright St
Urbana
IL  US  61801-3620
Primary Place of Performance
Congressional District:
13
Unique Entity Identifier (UEI): Y8CWNJRCNN91
Parent UEI: V2PHZ2CSCH63
NSF Program(s): OFFICE OF MULTIDISCIPLINARY AC,
COMPUTATIONAL PHYSICS,
Software Institutes
Primary Program Source: 01001920DB NSF RESEARCH & RELATED ACTIVIT
Program Reference Code(s): 026Z, 077Z, 7569, 7925, 8004
Program Element Code(s): 125300, 724400, 800400
Award Agency Code: 4900
Fund Agency Code: 4900
Assistance Listing Number(s): 47.070

ABSTRACT

The cyberinfrastructure needs for gravitational wave astrophysics, high energy physics, and large-scale electromagnetic surveys have rapidly evolved in recent years. The construction and upgrade of the facilities used to enable scientific discovery in these disparate fields of research have led to a common pair of computational grand challenges: (i) datasets with ever-increasing complexity and volume; and (ii) data mining analyses that must be performed in real-time with oversubscribed computational resources. Furthermore, the convergence of gravitational wave astrophysics with electromagnetic and astroparticle surveys, the very birth of Multi-Messenger Astrophysics, has already provided a glimpse of the transformational discoveries that it will enable in years to come. Given the unique potential for scientific discovery with the Large Hadron Collider (LHC) and the combination of the Laser Interferometer Gravitational-wave Observatory (LIGO) and the Large Synoptic Survey Telescope (LSST) for Multi-Messenger Astrophysics, the community needs to accelerate the development and exploitation of deep learning algorithms that will outperform existing approaches. This project serves the national interest, as stated by NSF's mission, by promoting the progress of science. It will push the frontiers of deep learning at scale, demonstrating the versatility and scalability of these methods to accelerate and enable new physics in the big data era. Because these methods are also applicable to many other parts of our national and global economy and society, this work will positively impact many fields. The students and junior scientists to be mentored and trained in this research will interact closely with our industry partners, creating new career opportunities, and strengthening synergies between academia and industry. The team will share the algorithms with the community through open source software repositories, and through our tutorials and workshops the team will train the community regarding software credit and software citation.

In this project, the PIs will build upon our recent work developing high quality deep learning algorithms for real-time data analytics of time-series and image datasets, as open source software. This work combines scalable deep learning algorithms, trained with TB-size datasets within minutes using thousands of GPUs/CPUs, with state-of-the-art approaches to endow the predictions of deterministic deep learning models with complete posterior distributions. The team will also investigate the use of Field Programmable Gate Arrays (FPGAs) to accelerate low-latency inference of machine learning algorithms to minimize the demands of future computing, which is a central goal for Multi-Messenger Astrophysics and particle physics. The open source tools to be developed as part of these activities will be readily shared with and adopted by LIGO, LHC, and LSST as core data analytics algorithms that will significantly increase the speed and depth of existing algorithms, enabling new physics while requiring minimal computational resources for real-time inferences analyses. The team will organize deep learning workshops and bootcamps to train students and researchers on how to use and contribute to our framework, creating a wide network of contributors and developers across key science missions. The team will leverage existing open source and interactive model repositories, such as the Data and Learning Hub for Science (DLHub) at Argonne, to reach out to a large cross-section of communities that analyze open datasets from LIGO, LHC, and LSST, and that will benefit from the use of these technologies that require minimal computational resources for inference tasks.

This project is supported by the Office of Advanced Cyberinfrastructure in the Directorate for Computer & Information Science & Engineering and the Division of Physics in the Directorate of Mathematical and Physical Sciences.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

PUBLICATIONS PRODUCED AS A RESULT OF THIS RESEARCH

Note:  When clicking on a Digital Object Identifier (DOI) number, you will be taken to an external site maintained by the publisher. Some full text articles may not yet be available without a charge during the embargo (administrative interval).

Some links on this page may take you to non-federal websites. Their policies may differ from this site.

(Showing: 1 - 10 of 13)
Chaturvedi, Pranshu and Khan, Asad and Tian, Minyang and Huerta, E. A. and Zheng, Huihuo "Inference-Optimized AI and High Performance Computing for Gravitational Wave Detection at Scale" Frontiers in Artificial Intelligence , v.5 , 2022 https://doi.org/10.3389/frai.2022.828672 Citation Details
Chen, Zhuo and Huerta, E. A. and Adamo, Joseph and Haas, Roland and OShea, Eamonn and Kumar, Prayush and Moore, Chris "Observation of eccentric binary black hole mergers with second and third generation gravitational wave detector networks" Physical Review D , v.103 , 2021 https://doi.org/10.1103/PhysRevD.103.084018 Citation Details
Gupta A., Huerta E. "Deep Learning for Cardiologist-Level Myocardial Infarction Detection in Electrocardiograms." 8th European Medical and Biological Engineering Conference , v.80 , 2020 https://doi.org/10.1007/978-3-030-64610-3_40 Citation Details
Habib, Sarah and Ramos-Buades, Antoni and Huerta, E A and Husa, Sascha and Haas, Roland and Etienne, Zachariah "Initial data and eccentricity reduction toolkit for binary black hole numerical relativity waveforms" Classical and Quantum Gravity , v.38 , 2021 https://doi.org/10.1088/1361-6382/abe691 Citation Details
Huerta, E. A. and Allen, Gabrielle and Andreoni, Igor and Antelis, Javier M. and Bachelet, Etienne and Berriman, G. Bruce and Bianco, Federica B. and Biswas, Rahul and Carrasco Kind, Matias and Chard, Kyle and Cho, Minsik and Cowperthwaite, Philip S. and "Enabling real-time multi-messenger astrophysics discoveries with deep learning" Nature Reviews Physics , v.1 , 2019 10.1038/s42254-019-0097-4 Citation Details
Huerta, E. A. and Khan, Asad and Davis, Edward and Bushell, Colleen and Gropp, William D. and Katz, Daniel S. and Kindratenko, Volodymyr and Koric, Seid and Kramer, William T. C. and McGinty, Brendan and McHenry, Kenton and Saxton, Aaron "Convergence of artificial intelligence and high performance computing on NSF-supported cyberinfrastructure" Journal of Big Data , v.7 , 2020 https://doi.org/10.1186/s40537-020-00361-2 Citation Details
Huerta, E. A. and Khan, Asad and Huang, Xiaobo and Tian, Minyang and Levental, Maksim and Chard, Ryan and Wei, Wei and Heflin, Maeve and Katz, Daniel S. and Kindratenko, Volodymyr and Mu, Dawei and Blaiszik, Ben and Foster, Ian "Accelerated, scalable and reproducible AI-driven gravitational wave detection" Nature Astronomy , 2021 https://doi.org/10.1038/s41550-021-01405-0 Citation Details
Joshi, Abhishek V. and Rosofsky, Shawn G. and Haas, Roland and Huerta, E. A. "Numerical relativity higher order gravitational waveforms of eccentric, spinning, nonprecessing binary black hole mergers" Physical Review D , v.107 , 2023 https://doi.org/10.1103/PhysRevD.107.064038 Citation Details
Khan, Asad and Huerta, E.A. and Das, Arnav "Physics-inspired deep learning to characterize the signal manifold of quasi-circular, spinning, non-precessing binary black hole mergers" Physics Letters B , v.808 , 2020 https://doi.org/10.1016/j.physletb.2020.135628 Citation Details
Khan, Asad and Huerta, E. A. and Zheng, Huihuo "Interpretable AI forecasting for numerical relativity waveforms of quasicircular, spinning, nonprecessing binary black hole mergers" Physical Review D , v.105 , 2022 https://doi.org/10.1103/PhysRevD.105.024024 Citation Details
Rosofsky, Shawn G. and Al Majed, Hani and Huerta, E. A. "Applications of physics informed neural operators" Machine Learning: Science and Technology , v.4 , 2023 https://doi.org/10.1088/2632-2153/acd168 Citation Details
(Showing: 1 - 10 of 13)

PROJECT OUTCOMES REPORT

Disclaimer

This Project Outcomes Report for the General Public is displayed verbatim as submitted by the Principal Investigator (PI) for this award. Any opinions, findings, and conclusions or recommendations expressed in this Report are those of the PI and do not necessarily reflect the views of the National Science Foundation; NSF has not approved or endorsed its content.

The project "Collaborative Research: Frameworks: Machine learning and FPGA computing for real-time applications in big-data physics experiments" has made significant contributions to the field of Multi-messenger Astrophysics. At the beginning of this project, AI had only been used in exploratory studies to address computational challenges in Multi-messenger Astrophysics. By the end of the project, the principal Investigators (PIs) and their students had developed state-of-the-art, open-source, physics-inspired AI models that reduced the time-to-solution by several orders of magnitude. They collaborated with other NSF-funded colleagues to publish their AI models on the Data and Learning Hub for Science (DLHub), making them easily accessible to the broader research community. The PIs also worked with other NSF-funded teams to integrate their AI models hosted on DLHub with NSF-funded High-Performance Computing (HPC) platforms and used Globus and Globus Compute to automate the processing of open-source gravitational wave data hosted at the Gravitational Wave Open Science Center.

The sophistication and complexity of the AI models developed in this project have increased over time. Initially, the models were based on the assumption of non-rotating, compact sources, which significantly simplified their development. This is because such compact binary sources, such as two colliding black holes, can be described using only two parameters - their masses. However, as the team developed more realistic AI models to search for spinning compact binary sources with elliptical orbits, they had to create scientific software to train the AI models using millions of modeled waveforms representing these astrophysical scenarios. This was necessary because these compact sources describe a high-dimensional parameter space, including masses, spin vectors, ellipticity, and more, which required dense sampling to accurately capture their physical properties.

By the end of this project, the team had developed novel computational frameworks capable of processing months-long gravitational wave datasets from the Gravitational Wave Open Science Center in just seconds using physics-inspired AI ensembles. They also utilized scientific data infrastructure, such as Globus and Globus Compute, to connect these AI models with HPC systems, automating and accelerating the processing of gravitational wave data using hundreds of GPUs. These improved AI models have been made available to the community through DLHub and have been used over 100,000 times. 

The team has also made innovations by developing the first class of AI models capable of forecasting the merger of neutron stars and the merger of stellar mass black holes with neutron stars.

In addition, the team has developed new AI methodologies to accelerate the computational modeling of compact binary sources, with a focus on multi-scale and multi-physics systems that can be described by magnetohydrodynamics equations. These AI models have attracted significant interest from the user community, leading NVIDIA to invite the team to upload their models to the NVIDIA Modulus platform.

Through this project, the PIs mentored several graduate students, resulting in the completion of three PhD theses, with two more PhD theses currently underway, as well as two Master’s theses. PIs also mentored several undergraduate students, many of them have gone on to pursue graduate studies in the US and abroad, while others have entered the workforce, primarily in the tech and finance industries in Silicon Valley and Wall Street.


Last Modified: 02/16/2024
Modified by: Volodymyr V Kindratenko

Please report errors in award information by writing to: awardsearch@nsf.gov.

Print this page

Back to Top of page