High-valued domain applications such as image recognition demand both fast innovation and high execution speed. For example, the methods for image recognition using deep neural networks evolve very quickly and require productive development environments for domain experts who may not be computer science experts. Geoffrey Hinton’s team won the 2012 ImageNet competition by training a deep neural network with 1.2 million images. Since then, many new algorithm innovations have been proposed for significant improvement over the version from Hinton’s team. Furthermore, the success in applying deep neural networks to image recognition has ignited a lot of research on applying deep neural networks to other areas such as speech recognition and natural language process, where traditional approaches have not been successful for decades.

At the same time, the research process for these methods require fast turnaround time of experiments that involve training deep neural networks using millions of images. Training of deep convolution neural networks involves fine-grained parallel computation within each layer but is constrained by data dependencies from one layer to the next. This is why traditional scale-out schemes based on clusters have not been successful. The best hardware for the training process has been tightly coupled multi-GPU systems that support extremely fast synchronization among a very large number of cooperating fine grained threads. Even with these systems, each such experiment can take weeks to complete. The creation of an effective algorithm can take many rounds of experiments. Dramatically more efficient implementation of candidate algorithms and thus much faster turnaround time of experiments can have a significant impact on the progress of the field. 

The state of the art in developing and implementing high-valued domain applications is based on application frameworks. For example, in machine learning, developers typically build on library frameworks such as Caffe, Torch, and TensorFlow. The library functions of these frameworks are implemented for CPUs, GPUs, and other accelerators. However, in the current practice, it takes tremendous amount of effort and time to bring up the implementation of an existing training method for a new compute device. It takes even more effort and latency to introduce a new method that needs to be implemented for the existing devices.  Individual kernels in each method needs to be hand-optimized for a new architecture. Efficient arrangement of the execution of kernels with respect to each other in a methods to take advantage of the memory hierarchy and interconnect capabilities requires even more effort.

Experience from the CUDA and Heterogeneous System Architecture (HSA) shows that architecture support can significantly reduce the cost of implementing application functions on heterogeneous parallel devices. Unified address space, user-level command queues, re-optimizable intermediate representations, and coherent shared memory are among the frequently cited system architecture features that reduce the barrier of implementation. However, little has been done in the architecture of the compute devices to explicitly lower the barrier of implementation.

On the compiler side, much progress has been made in C++ and OpenCL compilation to provide efficient code optimization, scheduling, and generation for a given algorithm. However, there has been little work on the support of specifying alternative algorithms that can achieve the  same application-level results but different levels of efficiency on different hardware types and hierarchies. Some recent work on DSL have shown promise in image processing. However, one needs to wrestle with the burden on the developers to learn multiple languages. A more generic extension to the existing C++ or Python language to allow more compiler support may be a more realistic long-term solution. 

The idea of the workshop is to bring together architecture and compiler researchers to discuss and debate on the potential approaches to accelerating the innovation in high-valued domain applications. The goal is to stimulate an in-depth discussion of what each can potentially achieve and how much more joint architecture and compiler approaches could accomplish. The product of the workshop is a report on the recommendations on each approach based on in-depth discussions of leading researchers from compilers, architecture, and important application domains.

Last Modified: 03/27/2020
Modified by: Lawrence Rauchwerger