The Clowder CSSI project developed a new version of the Clowder research data management platform to support convergent research across multiple research disciplines while expanding the open source community and user base. Much of the data needed for today's science is highly diverse and increasingly large in size. Managing, sharing, curating, and analyzing that data requires software, particularly because reproducibility benefits from programmability. Clowder provides an end user framework that is customizable to any discipline and scalable to modern day big data requirements. Clowder continues to bring together data and metadata management, information extraction, data visualization, social curation, and data sharing under one open source framework. Its ability to let users define their metadata fields, bring their own algorithms and pipelines, develop web based visualization, while scaling to very large datasets, in one environment provides a unique offering in the realm of data management for research.

As part of this project, we developed a new version of the software stack based on a decade plus of use and development of Clowder to support communities such as biology, geoscience, materials science, crop science, civil engineering, social science, and the humanities with support from NSF, ONR, NARA, and other federal and state agencies. This version 2 (v2) was developed from scratch using modern technologies such as Python, Typescript, and React.js to provide a much improved user experience and make it easier for the community to contribute to the codebase. 

Clowder v2 provides a brand new modern look based on Google’s Material Design system and a large number of improvements. For example, users can now version files and datasets and metadata can be associated with the specific versions. This lowers the clutter within the system and makes it easier for the researcher to update data and metadata. We have added brand new ways to share data with collaborators by introducing the concept of user groups and letting users enforce access on a dataset level. Machine metadata created by information extractors is now clearly separated from user defined metadata, with improvements on how the two are visualized and defined. Automated triggers for information extractors can now be defined using a generic query language as opposed to the original mime type implementation. This means that we can not only define what extractor will be automatically executed when a file of a particular type is uploaded, but also have more refined rules such as, a specific upload time, the pattern in the file name, or of a specific file size.

To broaden the community, we developed an online Webinar series, an in person community workshop and an online workshop, a hybrid hackathon, regular online dev meetings, and maintain an active Slack workspace. The webinar series included ten 1 hour long live presentations. The All Paws in person community workshops was a day long workshop co-located at PEARC 2019 with 45 participants. The online All Paws workshop in 2021 was 3 days long  online and had 72 participants. The hybrid hackathon was attended by 22 people. 

Throughout the life of the project the team has engaged with the material science community through the 4CeeD effort, the geoscience community through the Critical Interface Network (CINet) Critical Zone Observatory, the urban science community through the SMU partnership, the plant phenomics community through the TERRA-REF effort, the permafrost science community through the Permafrost Discovery Gateway project. These use cases have also generated positive outcomes. For example, the urban science use case discovered the presence of infrastructure deserts in Chicago and Dallas, which are low-income areas with significantly worse neighborhood infrastructure than other areas. Their findings were cited in City of Dallas’ Economic Development Policy and Economic Development Incentive Policy, the Dallas Housing Policy 2033, and 15 news stories. New areas of Clowder adoption and projects include NLP for literature mining of medical manuscripts, cultural heritage, analysing data from sensing devices for monitoring infants, managing microscope gigapixel images from archeological sites, 3D reconstruction of digital artifacts using photogrammetry, Arab American studies, cyberinfrastructure for deploying AI pipelines to the hybrid cloud, particle detection data management, HPC integration for particle imaging, and fossil pollen detection. Many of these efforts are still ongoing. Clowder version 2, a true open source data management platform that anyone is free to use and contribute to, will make it easier to adopt new use cases and extend the system based on new requirements over the foreseeable future.

Last Modified: 01/12/2025
Modified by: Barbara Minsker