This project, jointly undertaken by University of Washington, University of Michigan and Portland State University, explored new systems and algorithms aimed at delivering advanced database capabilities to the long tail of science -- the smaller labs and individual researchers who have increasingly extreme data requirements but limited IT staff.

We designed and deployed SQLShare, an open source, database-as-a-service platform that reduced the up-front costs of using database technology (installation, configuration, schema design, data ingest) to better fit a research environment where both questions and data sources change rapidly, and to lower barriers to adoption of data management technology.  The query and data workload collected over the multi-year lifetime of the project has exposed that scientists tend to analyze data in bursts -- a few complex queries over messy, short-lived datasets –- motivating new kinds of database systems, languages, and algorithms.  The workload has been released publicly for study by other researchers and represents a primary deliverable of this project.

To complement SQLShare, we developed the VizDeck automatic visualization system.SQLShare enables working with hundreds or thousands of distinct datasets, shifting the bottleneck of insight to visual analysis. VizDeck automatically recommends appropriate visualizations based on the statistical and structural properties of the data, allowing researchers to spend less time crafting visualizations and more time answering questions.  In our analysis, we found that the automatic visualization techniques in VizDeck significantly outperformed some commercial systems for visual data-exploration tasks. The VizDeck approach helped usher in a number of important projects in automatic visualization in both the database and visualization communities.

We developed the ReConnect and ReDiscover technologies to attack a different stage in the data pipeline: preparing data for publication and sharing in online databases, such as SQLShare.  We observed that scientists work with multiple variants and versions of datasets, often represented as spreadsheets, and that determining which datasets in a collection are the most appropriate to upload, combine, analyze and share is an impediment to sharing and the application of database technology. We posited that helping a researcher retroactively determine the relationships between datasets could facilitate the process. The ReConnect technology helps scientists determine the pairwise connections between datasets such as containment and complementarity. However, such pairwise analysis still leave a large burden on the researcher working with even a modest-sized collection of datasets, motivating the ReDiscover system, which applies data profiling, schema matching and machine learning techniques to automate the process of predicting relationships between datasets.

We developed the Senbazuru Spreadsheet Database Management System to focus on supplying database-style search-and-query capabilities over collections of heterogeneous spreadsheets.  The system allows automatic information extraction from a class of spreadsheets with complex implicit relational structures, extraction of brittle metadata by supporting efficient manual repair, and a suite of algorithms and UI tools on mobile and web platforms to support keyword search, structured query, and rich integration over the extracted information.

Last Modified: 12/28/2015
Modified by: Bill Howe