1 of 16

Clowder:

An Open Source Scientific Data Management Framework

​

​

​

Luigi Marini

Lead Research Software Engineer

National Center for Supercomputing Applications (NCSA)

University of Illinois at Urbana-Champaign

2 of 16

Clowder and Long Tail Data

Scalability

Extensibility

Data Size

Data Types

Clowder is a customizable and scalable data management framework to support any data format and multiple research domains

Heidorn, P. Bryan. "Shedding Light on the Dark Data in the Long Tail of Science." Library Trends, vol. 57 no. 2, 2008, p. 280-299. Project MUSE, doi:10.1353/lib.0.0036.

CLOWDER OVERVIEW

3 of 16

Clowder and Long Tail Data

4 of 16

A FRAMEWORK TO BUILD YOUR OWN DATA MANAGEMENT SOLUTIONS

Deploy on multiple backends

Customize and deploy your own instance

CLOWDER OVERVIEW

5 of 16

Upload, Share, Index Heterogenous Data

  • Create datasets of files, folders and metadata
  • Upload files, tag, license and organize them in datasets and collections
  • Download datasets and collections using BagIt
  • Selectively share datasets with collaborators and publish them to the Internet

​

Advanced Search

User management

File visualizations, License, tags

CLOWDER OVERVIEW

6 of 16

JSON Linked Data Metadata

User Specified

Machine Specified

API Machine Parsable

Searchable

Standard Vocabularies

6

CLOWDER OVERVIEW

7 of 16

Data Analysis

pyClowder library to simplify developing extractors in Python and R

Automatic Execution Events

  • File uploaded by Mime Type
  • File added/removed to dataset
  • Metadata added/removed to dataset

Provenance

Scalable

CLOWDER OVERVIEW

8 of 16

Data Visualizations

Vega.js

CLOWDER OVERVIEW

9 of 16

Custom Clients and Backends

  • Simple OpenAPI web API with user keys
  • Almost all functionality available in the Web UI is available in the Web API
    • Extractors use the Web API to write back results
    • Projects can create custom clients specifically tailored to their use cases
    • Data pipelines can leverage the API for custom workflows

CLOWDER OVERVIEW

10 of 16

TERRA-REF

ARPA-E TERRA-REF: A Reference Phenotyping System for Energy Sorghum

​

  • TB a night data transfers from Arizona (PBs of total data)
  • Image, Hyperspectral analysis extractors for phenotyping

CLOWDER OVERVIEW

11 of 16

4CeeD

NSF DIBBs T2C2: Timely and Trusted Curator and Coordinator Data (ACI-1443013)

​

  • Student data from material science labs
  • Microscopy image extractors
  • Support for diverse data ingestion
  • Student outreach and education

CLOWDER OVERVIEW

12 of 16

Developing permafrost big imagery products & making them discoverable for knowledge-generation�

+

=

You on your web browser:

Thaw slumps

Coastal erosion

Compute:

NSF Navigating

the New Arctic

Awards #

1927872, 1927723, 1927729, 1927720, 1927920, & 2052107

13 of 16

Clowder v2 - Join the Development!

  • Looking for Software developers, students, UI/UX designers, researchers
  • https://github.com/clowder-framework/clowder2

13

13

CLOWDER v2

14 of 16

Thank you!

Questions?

NSF OAC-1835834, 1835543, 1835877

​

https://clowderframework.org

15 of 16

Permafrost disturbances using ML, Landsat, & Sentinel

Nitze et al. 2018, Nature Geoscience

Multi-cloud solution

  • Google Earth Engine
  • Google Cloud
  • NCSA Radiant (OpenStack)
  • NSF Delta Supercomputer

Working with IBM Research to simplify building AI/ML pipelines using

16 of 16