The modern era of scientific code development is both a golden age and a dark age. It is a golden age because we have machines that can operate at unprecedented levels of performance. Unfortunately, it is also a dark age because these machines are becoming increasingly hard to program and use. This is due in part to power becoming a limiting resource, in response to which energy-efficient GPU computational accelerator designs have been introduced that omit energy-consuming niceties such as caches and predictors, replacing these with a baroque memory model and a requirement that programs be coded to follow a large number of threads of execution. Such accelerator designs demand much more effort from programmers to properly use, and programs must be re-tuned for each succeeding accelerator generation.

This need to re-tune is much less of a problem for CPUs, because features such as large caches and branch predictors enable compilers to generate good code without having to know, for example, how many times a piece of data will be accessed. GPU accelerators replace large caches with several types of storage, such as a high-speed scratchpad memory. The decision on whether to use these specialized storage areas depends upon aspects of code execution that the compiler often cannot determine. As a result the burden is placed on the programmer to decide how to stage data.
Though making use of specialized storage and other GPU features is beyond the ability of current compilers for *any* type of program, it is feasible to do for specialized domains, including for what are called stencil calculations. Stencil calculations, are used to solve many important scientific and engineering problems, such as simulating black holes, neutron stars, exploring quantum cosmology, simulating fluids, and performing coastal simulations.

The goal of the project was to develop a stencil framework, Chemora, that would allow a physicist or some other domain expert to code a stencil simulation in what is called a domain-specific language (DSL), and then run it on a GPU-accelerated cluster, and have its performance rival that of hand-tuned code. The use of the DSL makes it much easier for Chemora to generate code since Chemora knows much more about the movement of data than could be determined for an unconstrained language. Chemora can use all the kinds of specialized memory appearing on recent GPU accelerators (including shared, constant, and texture memory), and can transform calculations into pieces that each comfortably fit on the accelerator device. Chemora transforms a calculation based on a performance model of the device, in some cases optimizing certain rearrangements, in other cases using the model to choose among multiple candidates. Chemora operates in part when a program is run, which is when key information such as input data and system characteristics are first available. It takes advantage of this to generate highly efficient code.

Chemora can generate efficient code for several GPU generations. Chemora has been updated as new accelerators become available. Programs coded in Chemora's DSL enjoy good performance on the new devices without any effort required on the part of the original programmers.

The driver application for Chemora is a black hole simulation based on Einstein's equations. The application operates on a large number of quantities, something which introduces problems not encountered with simpler code. To accommodate this Chemora's model considers factors ignored by others, such as what is called register pressure. As a result, Chemora can efficiently run applications that may overwhelm other systems.
Future work will bring the Chemora research to production level, enabling scientists to make better use of computational resources and make new kinds of science possible in many areas that benefit humanity both directly and indirectly.

Last Modified: 11/29/2017
Modified by: Steven Brandt