1 of 14

STAGE Architecture Overview and User Narrative Breakout Prep

We have two core user narratives for WP2 but many, many for WP3
What’s lost in WP3’s user narratives is an architectural view of the system we’re building as a whole. We each see a piece of the overall system but, perhaps, not the whole.
Can we share the individual team’s vision of the architecture of DataSTAGE and, as a result, better understand what we’re collectively building together?

2 of 14

Proposed Activity

[Before F2F] Break into Element teams
[Before F2F] Each team draws out their understanding of the architecture of the system (what components, which group builds those components, which components “talk”) by Oct 2019
We have each team walk through their architectural diagram

C, Ca, He, Xe

Finally, we create a consensus diagram that shows the components from various groups and how they relate

3 of 14

Brainstorming

Let’s brainstorm in this shared document

https://bit.ly/2ERUDye �

No wrong answers, let’s collect ideas.�

Goal: how systems relate to each other �(not the details inside your system).

4 of 14

Xenon

Project Management

User Management

Authentication & Authorization

System Monitoring

Usage Logging

Notification Service

Backup Service

Billing Management

Web Application

API

Task Execution API

Data/Metadata Service

Cloud Storage & Compute

Resource Manager

Core Platform Infrastructure

Data Infrastructure

Independent Core Services

Task Execution Infrastructure

Task Scheduler

Job Management Layer

Orchestration Layer

5 of 14

FAIR4CURES

6 of 14

DataSTAGE

Powered by Seven Bridges

7 of 14

Science Applications aka Tools to enable the science: Science applications are user facing elements. Our process has been to deploy these as containers to DockerHub.

Notice the levels of application complexity

Bronze: Jupyter and RStudio are single container applications.
Silver: Cornerstone requires multiple cooperating containers to provide a user’s workspace.
Gold: Hail may require a user configured, potentially large number of containers to provide a workspace

User interface infrastructure is user facing in some capacity but not a vertical science application.

Platform Tools and App Infrastructure:

Jenkins provides our continuous integration environment with automated build, test, and deployment.
Anchore is a container security scanning solution we’re evaluating for analyzing container vulnerabilities.
Prometheus monitors the health and performance of the cluster.
OrthaNC is the image database used by several Dicom viewers. Perhaps we should treat things like it as “Stage Infrastructure”
Similar story for Spark. Packages like Hail run on Spark. It would be nice if applications could rely on Spark “just being there” as a platform service.

Container Orchestration. This is the layer that manages the lifecycle of multiple containers talking to one another to get work done for users.

Kubernetes is, by far, the leading solution for multi-container orchestration in a production environment.
We expect to use its API to dynamically deploy pods - sets of cooperating containers.
And we hope to use its NetworkPolicy capabilities to ensure an appropriate level of security.
A major operational challenge is to get this approach validated from a FISMA/ATO perspective to allow us to proceed to deployment.

We use Kubernetes as a compatibility layer across physical (and virtualized) environments.
We’re interested in a conversation with ATO’d stacks on the prospect of moving towards uniformity to make cross stack deployment manageable.

Integrating Geno-Pheno Search in Gen3 infrastructure

- So far the Geno-Pheno Search is a standalone demo. Now we are working on integrating this tool into Gen3.

- Initial requirements analysis reveals that this tool will rely on Windmill, Peregrine and Sheepdog services from Gen3.

- And it would also have high reliance on Monarch Knowledge Graph for ontology based queries.

8 of 14

PIC-SURE API

Part 1) Phenotypic data preparation (before an investigator logs in)

Part 2) Phenotypic query in real time by an investigator across platforms

decrypt

TOPMED Data Coordinating Center�Harmonization process

platform

Real time synonym search

Carbon

9 of 14

tranSMART

PIC-SURE �User Interface

i2b2 Core

PIC-SURE �Auth Micro-app

PIC-SURE �API V2

Fractalis

PIC-SURE HPDS

User Interfaces

Backend Services

Datastores

AuthN / AuthZ

Monitoring

AWS Account

Data Flow

Auth Flow

ETL Client

A) feasibility queries & cohort builder

B) Exploration

to generate hypothesis

C) Analysis

DataStore

Relational DB

Oracle + MySQL

HPDS

All logs ingested by Splunk

i2b2/tranSMART

platform

Carbon

10 of 14

DataStore

Relational DB

MySQL

HPDS

ETL Client i2b2/TM 18.1

Data Types

Clinical

Registries

Exome

Genome

i2b2/tranSMART

platform

Carbon

11 of 14

Ca+ Architecture

TOPMED WORKFLOWS

TERRA WORKSPACE

CROMWELL WORKFLOW ENGINE

JUPYTER INTERACTIVE ANALYSIS

WINDMILL DATA EXPLORER

U CHICAGO�Indexd

ORIGINAL METADATA

Metadata

harmonization across TOPMed datasets

Load by reference with GUIDs

ORIGINAL DATA FILES

DOCKSTORE TOOL REPOSITORY

Workflows via TRS

Metadata

via BDBag

Data files via DOS

Workflows

relevant to the

TOPMed community

AUTH

12 of 14

13 of 14

14 of 14

Next Steps

Diagram which component talks to which
Longer term views

Diagram that shows the ultimate destination… STAGE
Next one

User flows

Multiple flows
Not every component in every flow
Turn in to tutorials ultimately

Data flows

How data flows through the system

Apps and deployment process into ATO’d systems
Show security boundaries and how components are deployed