The nature of science is changing – it is increasingly limited by our ability to analyze the large amounts of complex data generated by our instruments and simulations: we see the emergence of Jim Gray’s “Fourth Paradigm” of science. Computers themselves are becoming the source of a lot of new data – the sizes of the largest numerical simulations of nature today are on par with the experimental data sets. This is not simply a computational problem, but rather requires a fresh look, and a holistic approach. We need to combine scalable algorithms and statistical tools with novel hardware and software solutions, like a deep integration of GPU computing with database indexing and fast spatial search capabilities. We propose to build a new kind of instrument, a ‘Data-Scope’, that is capable of observing very large amounts of scientific data, with unique features in its design.

In sciences today tackling data-intensive problems at the 5-10TB scales is easy: one can perform these analyses at a typical generic departmental computing facility. 50-100TB problems are quite difficult, but there are about 10-15 universities in the world that can analyze such data sets. When one needs to deal with a petabyte of data, there are less than a handful of places anywhere in the world that can address this challenge. At the same time there are many projects which are crossing over the 100TB boundary today. Astrophysics, High Energy Physics, Environmental Science, Computational Fluid Dynamics, Genomics and Bio­informatics are all encountering data challenges in the several hundred terabyte range and beyond – even within a single university. The large data sets are here, but the off-the-shelf solutions for their analyses are not!

The Data-Scope instrument has unique capabilities: it combines about 6.5 Petabytes of storage with a sequential IO bandwidth exceeding 500GBytes/sec and 120 Teraflops of GPU computing. In order to keep the cost of the instrument down, and its performance and storage capacity very high, all at low power consumption, there must be tradeoffs. The Data-Scope was tuned to provide extreme data analysis performance over petabytes at the expense of some generic features. It is a highly specialized tool to study data, a microscope for data: a “Data-Scope”, which is why we consider this to be more similar to a research instrument rather than a traditional computational facility. Since its commissioning it has enabled certain analysis tasks that would have been extremely difficult otherwise. Two of JHU’s Nobel Laureates and their students are among the early users of the Data-Scope.

This new, data-intensive nature of science is becoming increasingly important by the day.  There is a similar vacuum in our abilities to handle large data sets now as there was in the 90’s when the concept of the BeoWulf cluster emerged. Many universities and scientific disciplines are looking for a new template that would enable them to address PB scale data analysis problems. In providing an inexpensive hardware and software architecture, we feel that we can substantially accelerate the development of data-intensive science in the whole country. In order to accelerate the acceptance of the proposed approach we will collaborate with researchers across many different disciplines and across many different institutions nationwide (Los Alamos, Oak Ridge, UCSC, NMSU, UW, UC, UIC, UIUC). The Data-Scope is hosting public services on some of the largest data sets in astronomy, and fluid mechanics, both observational and simulated. Our public turbulence database services (close to 500 Terabytes) have delivered over 10 trillion data points to the world. Students and postdoctoral fellows using the Data-Scope are gaining a substantial career advantage – these will be the job skills of the 21^st<...

Please report errors in award information by writing to: awardsearch@nsf.gov.