The Scalable Cost-Aware Cloud Infrastructure Management and Provisioning (SCRIMP) project developed new methods and open-source software for profiling, predicting, and provisioning cloud resources. The project created new methods for profiling application performance across diverse cloud resources, new techniques for predicting cloud costs in preemptible and volatile cloud markets, and developed a user-facing provisioning service that leverages profiling and prediction models to enable users to quantify, balance, and optimize tradeoffs in terms of application performance (e.g., accuracy) and costs (e.g., time, budget, and energy consumption). 

 The SCRIMP profiling framework is designed to automatically and efficiently profile application performance on different cloud and edge resources. The framework is able to capture a multidimensional notion of cost, for example, cloud computing costs, startup costs, data movement costs, and computing resources used. It also considers various measures of performance, such as result quality or application accuracy. Such flexibility is crucial, for example, in machine learning training, as the profiling framework can capture model accuracy and thus enable downstream exploration of the tradeoff between training time and model accuracy. The profiling framework can automatically explore wide input parameter spaces for applications, investigating different parameterizations to estimate performance and costs. The profiling framework uses active learning/experiment design methods to reduce profiling costs via targeted samples across the search space.  Experiment results show that the profiling framework can accurately predict costs on heterogeneous distributed resources. 

The project created new methods to predict cloud computing costs based on analysis and modeling of cloud markets. Analysis of AWS spot market dynamics contributed to  understanding of the “predictability” of the spot market over a long period of time. This analysis highlighted important changes to the market that altered the prediction dynamics, and ultimately showed that recent changes dramatically simplified the market making it far less dynamic than it was previously. As a result, even simple statistical methods can accurately forecast prices into the future. The project developed and evaluated neural network and statistical models to forecast prices. Evaluation showed that even ARIMA models achieved an order of magnitude improvement in accuracy when applied to recent spot market data compared to historical data. These prediction methods make it easy to forecast future prices and when used with real applications can significantly reduce costs and resource waste. 

Finally, the project created a new service called DELTA that combines these profiling and prediction methods to automatically provision and schedule execution of user workloads across distributed and  heterogeneous resources. DELTA enables applications to be executed without requiring users to consider the complex costs that exist in cloud and edge systems (e.g., provisioning delays, transfer costs, container deployment time). DELTA implements an extensible architecture in which different predictors and scheduling algorithms can be integrated to provide dynamically evolving estimates of costs on different resources. These estimates can be used to determine the most appropriate location for execution. Experiments across diverse resources, including cloud and edge devices, showed that DELTA  can significantly reduce workload makespan when compared with a strategy that selects the fastest resource. 

Last Modified: 02/28/2024
Modified by: Kyle Chard