High Performance Computing (HPC) systems are widely used in fields as diverse as weather modeling, financial predictions, fluid dynamics, and big data searches, and are opening the doors to new discoveries. For instance, the field of genomics relies heavily on very large supercomputers. Because of genomics, we have new drugs, ways of diagnosing disease, and crime investigation techniques. But the energy costs of operating HPC systems, whether it be supercomputers, data centers, or clusters of machines, are becoming prohibitive as HPC systems evolve. As an example, the K supercomputer in Japan consumes enough energy to power 10,000 homes, at a cost of $10 million/year. The petascale Yellowstone supercomputer at NCAR in Wyoming, USA has an energy expenditure of $2 million/year. Even smaller terascale HPC clusters, such as those at Colorado State University (CSU) and Oak Ridge National Lab (ORNL), have significant computational and cooling energy costs, ranging from $20K-200K per year. These energy costs impose a huge monetary burden on the scientific community, taking HPC systems out of the reach of those who have the greatest potential to make groundbreaking discoveries that can benefit society. Prior efforts to improve energy-efficiency in computing are unfortunately either not applicable to large HPC platforms or ignore important facets of the problem, such as cooling/thermal costs and uncertainties that often surface in HPC platforms at runtime.

The overarching theme of this proposal has been to devise a new software-based resource management framework that can intelligently manage the execution of applications on large-scale HPC platforms, while minimizing the energy needed for computation and cooling. The fundamental innovation that has emerged from this project is the discovery of the complex relationship between cooling and computation energy in HPC platforms, and its characterization using stochastic performance, robustness, and power models derived from real-world data (from terascale and petascale HPC systems). The insights from these models have guided the design of new strategies to co-optimize computing performance, robustness, computation power, and cooling power in large-scale HPC platforms. These strategies have further benefitted from new models that have been developed for 1) quantifying the impact of interference in shared memory and network subsystems; 2) fast real-time thermal characterization; and 3) cooling energy costs and capacity.

Rigorous experimental analysis has shown that the developed models and framework significantly outperform the best known prior efforts to quantify and optimize energy usage in HPC platforms. This project has also contributed to the integration of energy-efficient resource management in production HPC systems at ORNL and the Department of Defense (DoD). The research contributions ultimately represent unique and valuable solutions to overcome the energy challenge facing the design of future HPC platforms. The innovations from this research have been widely disseminated through over 50 peer-reviewed scientific journal/conference publications, as well as several invited industry and conference seminar talks, keynotes, and tutorials. The technical outcomes of this project have thus made significant and lasting contributions towards the goal of meeting current national needs for energy-efficient and cost-effective HPC systems.

Beyond the technical objectives accomplished, this project also has had an immense broader impact. The techniques developed as part of this project have the potential to be applied to a variety of computing and communication system environments. As an example, the algorithms and models for energy-efficient HPC resource management from the project were successfully applied to solve multi-objective resource management problems in manycore electronic chip design. Several students have been fully or partially supported by this project. A total of eight Ph.D. students, two post-doctoral fellows, three M.S. students, and seven senior undergraduate students have conducted research with the faculty as part of this project. As part of K-12 outreach, four high school students also have been provided opportunities to work with the senior students on this project, and learn about the exciting opportunities in computer engineering. By exposing these students to the diverse aspects of modeling and analysis, optimization algorithms, emerging hardware, and software applications, and disseminating the developed ideas and outcomes via curriculum enhancements at CSU, the proposed research has also significantly contributed to an agile high-tech workforce that will maintain continued USA leadership in technological innovation. 

Last Modified: 01/26/2018
Modified by: Sudeep Pasricha