High-throughput applications measure their performance in computational throughput using distributed resources and are vital for scientific discovery. These applications are also increasingly complex, combining multiple components: data generation; data collection and merging; data pre-processing and feature extraction; data analysis and modeling; and data verification, validation, and visualization. Adverse conditions, such as heterogeneous, unreliable architectures at all scales, including extreme scale, testing under uncertainty, black-box methods, and unexplainable algorithms, hinder the ability of scientists to generate robust science.  Robust science uses research methods that are scalable, reproducible, and trustworthy for generalizable solutions. 

There are three essential requirements to achieve robust science in high-throughput applications: 

Performance scalability: High-throughput applications must meet hardware and software performance expectations when executed, despite heterogeneous resources and large-scale systems. Performance scalability can be enhanced by using consistent metrics and methods to measure computing experiments and deploying rigorous scheduling and resource provisioning models to map tasks to available infrastructure efficiently.

Reproducibility: Scientists must be able to draw the same scientific conclusions using the knowledge encapsulated in the original computational experiment. In the ICERM report, this was referred to as confirmable research. Reproducibility can be accomplished by verifying and leveraging others? findings, supporting and exploring alternative methods, and explaining algorithms. 

Trustworthiness: Scientists must trust the technology, people, and organizations delivering their scientific discoveries.  Trust can be accomplished by providing software and data security solutions while supplying the necessary attributes for confidence in the scientist?s results and results from others.

These three requirements are the driving factors in any roadmap to pursuing robust science. Specifically, scientists should target performance scalability (both spatial and temporal) as a metric of success to meet the scalability requirements; correctness (by overcoming data corruption, faulty software, and system failures) as metrics of success to meet the trust requirements; and modeling accuracy at a range (e.g., through verification and validation) as a metric of success to meet the reproducibility requirements. These findings are the outcome of two virtual mini-workshops in February and May of 2021 called Virtual World Cafes (VWC) based on the world cafe method. The two VWC engaged application communities to share needs and recommendations through structured conversational processes. Participants were distributed across several breakout sessions in an online meeting, switching sessions periodically and getting introduced to the previous discussion at their new session by a session lead.

Overall, a successful roadmap to robust science for high-throughput applications builds on increasingly complex multi-modal workflows. The first step to success in delivering scientific discovery for these applications is to establish a vibrant next-generation community that works together to define, design, implement, and use robust solutions. The second step is to build those solutions to span five critical areas: architecture; systems; high-performance computing; programming models and compilers; and algorithms and theory. The last step comprises combining these areas into an integrated continuum through AI orchestrations, policies, and practices accessible to the newly created communities.

Last Modified: 01/23/2023
Modified by: Trilce Estrada-Piedra