Over the last 20 years, the amount of available scientific data has doubled approximately every year - every year adds more measurements of the universe than existed in all of human history. Computers and information technology have struggled to keep pace with this data avalanche, and researchers have had to develop new skills to make sense of it all. New approaches are needed to store, analyze, visualize, and publish data at such high volumes. Furthermore, data sets have grown too large to be downloaded and processed locally - science must be done online, close to the data, if it is to be done at all.

Building on 20+ years of experience that our team at Johns Hopkins University?s Institute for Data-Intensive Engineering and Science (IDIES) has garnered hosting and serving large astronomy data sets, we have created SciServer (www.sciserver.org), a full-featured ?science platform? that enables scientists to conduct data-driven research with the largest scientific datasets, entirely online.

In 2001, we debuted SkyServer (skyserver.sdss.org), the first online portal for data-intensive astronomy. SkyServer gave the entire world free access to Terabytes of data from the Sloan Digital Sky Survey (SDSS), an ongoing worldwide effort to make a three-dimensional map of the universe. SkyServer has been one of the most widely used public science portals, resulting in thousands of research publications and millions of students and citizen scientists learning science by doing science with real data.

SciServer greatly increases the capabilities of SkyServer and expands its content from astronomy to all sciences (hence the change in name).

SciServer provides a complete online environment for working with big scientific data sets, with features for researchers, educators, and data providers - all accessible through a single web destination. This includes:

• A versatile user interface accessible to novices and professional researchers alike (see Figs 1-2)

• Components supporting query, computation, visualization, and analysis of data sets in any field of science

• Server-side access to Petabytes of data, covering topics such as astronomy, oceanography, genomics, and materials science.

• The ability to store personal data sets online

• Mechanisms to support sharing of public and private datasets - promoting public datasets while keeping private datasets securely accessible only by designated members of a research team

• Educational tools for the next generation of data scientists and their instructors, designed to make sharing of datasets and activities as simple as possible for both in-person and virtual classroom environments

Like our previous projects, SciServer gives users the ability to submit free-form queries to Terabyte-scale databases, and to save data of interest to a personal database. SciServer Compute extends this functionality greatly with tools that allow users to write programs in a range of computer languages, which can access powerful scientific software libraries, without having to download, install, or configure anything. These programs run online on computers that have fast access to data servers, and all scripts can be executed in both interactive and batch mode. See Figure 3 for some examples.

Each SciServer user receives 10 GB of stable, backed-up private storage, plus access to Terabytes of shared scratch space for short-term storage of larger data products. Curators of large datasets who wish to disseminate their data publicly through SciServer can contact us and we will assist them to share their data, either with all SciServer users or with invited users or groups, with custom read/write/share permissions.

The full SciServer system launched in June 2018, and has grown steadily in breadth and scope.  We support more than 50 research projects across many different scientific disciplines as illustrated in Figure 4. New data sets include extensions to catalogues from SDSS and other astronomical surveys, and also data from the fields of medicine and bioinformatics, social sciences and even literature. SciServer also provides access to truly virtual data, produced by large computer simulations of the universe, ocean circulation and of materials science for example, that can be compared to "real" data on the same platform.

Separate instances of SciServer have been deployed at the Max-Planck Institute for Extraterrestrial Physics in Munich, where it supports data from the eROSITA X-Ray satellite, as well as at the National Institute for Standards and Technology (NIST), where it is being used in to investigate the effects of Hurricane Maria on Puerto Rico in 2017.

SciServer has been used in courses at JHU and at other sites such as the University of St. Andrews in Scotland.  An exciting new capability is to spin up a temporary SciServer instance in a commercial cloud, which can be used for courses or schools to manage their own users and data sets.

Most importantly, we support a community of over 8,000 users, including many educators who are using SciServer to teach data science and computational thinking to the next generation of scientists.

Last Modified: 08/12/2020
Modified by: Michael Rippin