Intellectual Merit:

This project resulted in a proof-of-concept prototype implementation of a universal data access and sharing building block for data-intensive applications, called OneDataShare. It provides optimization of end-to-end data transfer performance with novel application-layer models and algorithms, interoperability between heterogeneous data resources and endpoints, and the ability to accurately predict the data delivery time to decrease the uncertainty in real-time decision-making processes. The project resulted in over 15 peer-reviewed publications in major venues. Some of the most impactful innovative components of OneDataShare include:

(1) A novel dynamic parameter tuning algorithm based on historical data analysis and real-time background traffic probing, called HARP. Most of the previous work in this area is solely based on real-time network probing or static parameter tuning, which either result in an excessive sampling overhead or fails to accurately predict the optimal transfer parameters. Combining historical data analysis with real-time sampling lets HARP tune the application-layer data transfer parameters accurately and efficiently to achieve close-to-optimal end-to-end data transfer throughput with very low overhead. Instead of one-time parameter estimation, HARP uses a feedback loop to adjust the parameter values to changing network conditions in real time. The experimental analyses over various network settings show that HARP outperforms existing solutions by up to 50% in the achieved data transfer throughput. This work was published in IEEE Transactions on Parallel and Distributed Systems (TPDS) in 2018.

(2) Three novel algorithms for application-layer parameter tuning and transfer scheduling to maximize transfer throughput in wide-area networks. These algorithms use heuristic methods to tune the level of control channel pipelining (for small file optimization), the number of parallel data streams per file (for large file optimization), and the number of concurrent file transfers to increase I/O throughput (a technique useful for all types of files). The proposed algorithms improve the transfer throughput by up to 10x compared to the baseline and 7x compared to the state-of-the-art solutions. These algorithms were published in the Journal of Parallel and Distributed Computing (JPDC) in 2018.

(3) A novel two-phase application-layer throughput optimization model for big data transfers based on offline knowledge discovery and adaptive real-time tuning to ensure continuous performance guarantee and fairness among the contending transfers in the shared network. In the offline analysis phase of this model, historical transfer logs are mined to perform knowledge discovery about the transfer characteristics. The online phase uses the discovered knowledge from the offline analysis and real-time investigation of the network condition to optimize the protocol parameters. This novel approach is tested over different networks with different datasets and outperformed its closest competitor by 1.7x and the default case by 5x. It also achieved up to 93% accuracy compared with the optimal achievable throughput possible on those networks. This work was initially presented at IEEE BigData 2017 conference, and an extended version was published in the IEEE Transactions on Parallel and Distributed Systems (TPDS) in 2020.

(4) A novel solution for efficient and scalable metadata access for distributed applications across wide-area networks. This novel solution, dubbed SMURF, combines novel pipelining and concurrent transfer mechanisms with reliability, provides distributed continuum caching and semantic locality-aware prefetching strategies to sidestep fetching latency, and achieves scalable and high-performance metadata fetch/prefetch services in the Cloud. It incorporates the phenomenon of semantic locality awareness for increased prefetch prediction rate using real-life application I/O traces from Yahoo! Hadoop audit logs and proposes a novel prefetch predictor. By effectively caching and prefetching metadata based on the access patterns, this novel continuum caching and prefetching mechanism significantly improves the local cache hit rate and reduces the average fetching latency. During approximately 20 million metadata access operations from real audit traces, SMURF achieved 90% accuracy for prefetch prediction and reduced the average fetch latency by 50% compared to the state-of-the-art mechanisms. This work was published in IEEE Transactions on Parallel and Distributed Systems (TPDS) in 2022.

Broader Impacts:

This project resulted in the training and professional development of 31 graduate students (7 Ph.D. and 24 M.S.) and 11 undergraduate students through active participation in the research and development tasks of the project. In a half-day workshop, the basics of cloud computing, data-intensive computing, and the state-of-the-art OneDataShare data-sharing technologies were introduced to a group of high school students (mostly low-income inner-city kids) interested in STEM education. OneDataShare`s novel data transfer optimization, interoperability, and prediction services will drastically increase the end-to-end performance of data transfers and data-intensive applications in different domains which depend on data movement. The cloud hosted OneDataShare data transfer scheduling and optimization service has the potential to become a key component of the national data access and sharing infrastructure.

Last Modified: 01/04/2023
Modified by: Tevfik Kosar