The Cognitive Hardware And Software Ecosystem Community Infrastructure (CHASE-CI) project successfully created a private cloud of hundreds of affordable Graphics Processing Units (GPUs) networked together to facilitate development of next generation cognitive computing, which was being impeded by high costs and limited access to hardware.

Machine learning (ML) with large datasets, either static or streaming, is one of the fastest growing fields in computer science and engineering. The CHASE-CI project designed and propagated a strong research community infrastructure and human support system to address current challenges to ML research by supporting the ML discipline and its allied applications with hardware, system software, data, networking, lab notebooks, and workflows. It also has proven to be useful to researchers in computational media.

Leveraging the existing NSF-funded Pacific Research Platform (PRP), CHASE-CI created an initial community of more than 30 ML researchers and facilitated their research efforts by making available several hundred 32-bit GPUs and large amounts of storage for off-line computing of training weights or parameters from large numbers of data samples. CHASE-CI built a custom distributed cyberinfrastructure called Nautilus and made it compatible with the commercial clouds and NSF Supercomputer Centers, which have vastly increased their offerings of a variety of GPUs over the past three years. To date, the project has made more than 500 GPUs accessible to the ML community, and at least 103 active projects were using approximately 3,500,000 GPU hours in the Nautilus cluster in the final year of funding. The largest ML GPU users in the last year were exploring enhancement to a variety of techniques, including self-supervised, reinforcement, deep CNN, multi-task, and 3D scene learning. Applications of these improved ML algorithms were to robotics, self-driving cars, genomics, medical records, and COVID-19 forecasting.

To sustain the success and usability of the CHASE-CI infrastructure, an important goal of the project was to train university IT professionals to care for the developed GPU clusters and to encourage building of similar clusters of GPUs for student use in classes and labs. By the final year of the project, 30,000 students in UCSD courses had used a campus-funded 128-GPU Nautilus-cloned cluster. The Kubernetes Nautilus cluster developed for the PRP has also been seamlessly offered as a means for researchers to access cloud resources over high-speed networks such as CENIC and Internet2 that are connected to Amazon, Google, and Microsoft clouds. CHASE-CI provides partners with access methods to use these commercial clouds; some NSF supercomputers, Google, and Amazon Web Services now support Kubernetes orchestration. In addition, tools for measuring and monitoring scientific use of CHASE-CI resources were created that allow developers to tabulate and visualize quantitative information so that CHASE-CI resources can be continually compared to commercial cloud offerings and updated as needed to remain state-of-the-art.

Last Modified: 11/29/2021
Modified by: Larry L Smarr