Advances in technology have enabled us to collect large amounts of
data from all walks of life.  Managing and processing such large
volumes of data, or "Big Data", and gaining meaningful insights is a
significant challenge facing the Big Data community. This has
significant impact in a wide range of domains including health care,
bio-medical research, Internet search, finance and business
informatics, and scientific computing. As data-gathering technologies
and data-sources witness an explosion in the amount of input data, it
is expected that in the future massive quantities of data in the order
of hundreds or thousands of petabytes will need to be processed. Thus,
it is critical that data-intensive computing middleware (such as
Hadoop, Spark, and HBase) to process such data are diligently
designed, with high performance and scalability, in order to meet the
growing demands of such Big Data applications.

While Hadoop, Spark, and HBase are gaining popularity for processing
Big Data applications, these middleware and the associated Big Data
applications are not able to fully take advantage of the advanced
features on modern High Performance Computing (HPC) systems widely
deployed all over the world, including many of of the multi-Petaflop
systems in the XSEDE environment. Modern HPC systems and the
associated middleware (such as MPI and Parallel File systems) have
been exploiting the advances in HPC technologies (multi/many-core
architectures, RDMA-enabled networking, NVRAMs and SSDs) during the
last decade.  However, Big Data middleware (such as Hadoop, Spark, and
HBase) have not embraced such technologies. These disparities are
taking HPC and Big Data processing into "divergent trajectories".
This leads to the following broad challenges: "Can novel runtimes be
used to redesign Big Data middleware (such as Hadoop, Spark, and
HBase) to deliver performance and scalability on modern and
next-generation HPC systems?"

In this project, we have proposed new runtimes and re-designed the Big
Data middleware stacks to take advantage of modern HPC technologies
and systems. Challenges have been addressed in the following three
directions: 1) Designed novel communication and I/O runtime for Big
Data processing while exploiting the features of modern
multi-/many-core, networking and storage technologies; 2) Redesigned
Big Data middleware (such as Hadoop, Spark, and HBase) to deliver
performance and scalability on modern and next-generation HPC systems;
and 3) Demonstrated the benefits of the proposed approach for a set of
driving Big Data benchmarks and applications on HPC systems.  The
proposed designs have brought HPC and Big Data processing into a
"convergent trajectory".

Contributions were made in all three areas and evaluated with a range
of Big Data benchmarks and applications including - TeraGen, Sort,
TeraSort, TestDFSIO, GroupBy, PUMA, YCSB, CCIndex, HiBench,
CloudBurst, Deep Learning, and Astronomy.  Some highlights of these
results are:

* The new RDMA-HDFS design delivers a speedup of 2.3x for TeraGen of
  200 GBytes of dataset

* There is an improvement of 25% for 120 GB Sort using the new
  MapReduce over Luster

* The RDMA-Spark design over 1536 cores improves the HiBench PageRank
  time by 43%.

* An astronomy application using 65GB dataset is accelerated by 21%
  using the proposed RDMA-Spark design.

* Up to 2.4x speedup for the YCSB workload A with RDMA-based design
  for HBase.

The results of this research (new designs, performance results,
benchmarks, etc.) have been made available to the community through
RDMA-Hadoop, RDMA-Spark, and RDMA-HBase libraries and OSU HiBD
(High-Performance Big Data) benchmark suite (multiple versions and
libraries).  The latest versions of these libraries are currently
running on many large-scale InfiniBand and RoCE systems including SDSC
Comet. Currently, the HiBD libraries are being used by more than 275
organizations in 34 countries. The HiBD libraries and the associated
enhancements are being used by a large number of users of these
systems to accelerate Big Data applications.
 
In each of these releases, information about the tuned designs for
various components (HDFS, MapReduce, Spark, HBase, etc.) has been
shared with the HiBD user community through mailing lists. The
applications-based results have been made available to the community
through the "Performance" link of the HiBD project web page.  In
addition to the software distribution, the results have been presented
at various conferences and events through talks, tutorials, and
hands-on sessions.  Multiple Ph.D and Masters students have performed
research work and received their Ph.D and M.S. degrees as a part of
this project.

Last Modified: 12/26/2017
Modified by: Amitava Majumdar