Intro

Rok Roškar, Swiss Data Science Center, ETH Zürich

rok.roskar@sdsc.ethz.ch

12 September 2023

Slides: https://bit.ly/renku-hdbi-slides

2

Outline

3

Renku Hands-on Tutorial

​

What’s ahead?

​

What is Renku?

Renku enables “sustainable” data science

1. Done today with tomorrow in mind
2. Individual work benefits institution and community
3. All components form a functioning ecosystem

4

5

a story…

Development

Publication

Many practical and technical hurdles along the way from data to results…

6

Development

Publication

Option 2

Renku streamlines the behind-the-scenes work, so you can focus on the fun parts of data science!

The Knowledge Pyramid: questions & incentives

7

Individual researcher(s)

Knowledge

Organization

Working on Data

Lab, group, institute

Community, funding body, university, national org

I can’t reuse my teammate’s work because I don’t understand it.

I spend extra time re-running my own code

Sharing my code and data with others requires extra effort.

How can we make sure work is not wasted with each new student turnover?

How can I showcase my group’s work?

How can my work make efficient use of compute resources?

What work has been done in this domain?

How is funding money being spent?

How can we connect researchers within and across domains?

Addressing the points of friction bubbles knowledge upwards

What is Renku?

8

Code

9

Datasets

• Create datasets to easily reuse and share data across projects
• Use various backends: git-LFS, S3, Azure blob, local or network storage sources
• Combine with compute environments and analysis examples to ensure data can easily be used and reused
• Record pipelines that yield or consume datasets for full traceability

10

Create (choose storage), assemble, annotate, publish

Workflows

11

Capture workflow

Record as KG

Optimized storage

Reuse on various backends

toil on HPC

renku run my-analysis.sh

Additional via plug-ins

Automatically

record pipelines

12

Environment

• Easy access to shared compute and storage
• Containers for reproducibility and portability, templates for consistency
• Maintained library of images to keep things up-to-date; install apps, dashboards, desktops etc.
• Configurable access to resources
• Shared project data sources

For the user, there is NO vendor or technology lock-in

apart from git + docker

The Renku Stack

https://renkulab.io

command-line interface (CLI)

Hands-on part!

Go to: https://bit.ly/renku-hdbi-tutorial

16

17

Harness code & data versioning

Use the knowledge graph for <creativity>

Containerize your compute environment

Track & show file lineage via a workflow tool

Renku helps data scientists improve their computational skills, wherever they are on their journey.

Why use Renku?

What’s ahead?

18

Where is Renku used?

• Public instance at renkulab.io; several other smaller instances
• Primary use-cases:
• Teaching (courses, workshops)
• Small teams working on data analysis projects
• Showcasing of derived datasets and results
• Improving reusability of data products
• Used as a “connecting” piece (e.g. enabling collaborative access to data products from other platforms)

​

19

What’s ahead

• More flexible/easier “on-boarding” of users, projects, data
• More comprehensive overviews of where data is used
• Better integration with data providers
• Our goal is to make data “alive” – how can we do better? What would you imagine to be useful for your community?
• Renku as the “middle layer”/connector of code, compute, data
• Organizational/group views, “hub pages”

20

Organizational “hub” view

21

We want to hear from you!

🙋 Try out Renku

• renkulab.io - Public

​

📃 Renku Docs

​

❓ Run into a problem?

• Post on Discourse (our forum)
• Submit a bug report

​

💡 Feature Request?

• Discourse!

​

22