Project Outcomes Report

Background: We live in an era of big data where large amounts of data are generated in every walk of life. One of the major challenges we face in dealing with such a vast growth in data lies in creating methods to analyze these datasets and extract useful information from them. This project had the major goal of developing efficient computational techniques to analyze big data arising especially in the domain of biology. In particular, this project was aimed at developing effective computational algorithms for big data analytics. These algorithms should be suitable to be run on a single machine as well as multiple machines. The use of multiple machines (i.e., parallel computing) is vital for processing voluminous datasets.

Another important aspect of big data analytics is that the data may be very large and it may not fit in the main memory of a computer. In other words, the data may have to be stored in slower storage devices such as disks. In this case we have to ensure that the number of data accesses from the disks is minimized. Algorithms that minimize these data accesses are referred to as out-of-core algorithms. In this project we have focused on the development of novel parallel and out-of-core algorithms for solving numerous fundamental problems in biological big data analytics.

Intellectual Merit: We have developed parallel and out-of-core algorithms for solving a number of fundamental problems including motif search, string and sequence analysis, data compression, data linkage, correlational study between phenotypes and genotypes, finding the closest pair of points, feature selection, making a neural network sparse, and error correction in sequence data. These algorithms outperform the prior algorithms for the respective problems in terms of run times and accuracy. Our research results have been published in top-notch journals and conferences such as ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), Nucleic Acids Research, Bioinformatics, Journal of the American Medical Informatics Association (JAMIA), IEEE/ACM Transactions on Computational Biology and Bioinformatics (TCBB), International Conference on Neural Information Processing Systems (NeurIPS), ACM International Conference on Information and Knowledge Management (CIKM), SIAM International Conference on Data Mining (SDM), and IEEE International Conference on Data Mining (ICDM). We have published thus far 19 journal papers and 33 conference papers. In addition, we have submitted 4 journal papers and 2 conference papers for publication.

Broader Impacts: Given the generic nature of the algorithms we have developed, they can be applied for different domains. Examples include Materials Science, Business, Physics, etc. For instance, our record linkage algorithms should be of use to healthcare providers. We expect that the society at large could benefit soon from the outcomes of our research. All the software developed in this project are open source and have been freely released to the public through appropriate forums such as GitHub. 16 graduate students (4 of them being women) have worked on this project in the duration of the project. In addition, several undergraduate students have also worked. These students have received excellent training in big data and conducting research. Some of these students have graduated with a Ph.D. and joined the industry (in Google, Amazon, etc). They are expected to apply the knowledge they have gained to solve real life problems. We have introduced the findings of this project in relevant courses as well. As a result, more students have been exposed to the topic of our research.

Last Modified: 09/13/2019
Modified by: Sanguthevar Rajasekaran