Jupyter Iceland Project
Sigurður Detlef Jónsson
Implementation for the Icelandic Research-E Infastructure IREI
IREI
Agenda
2/20
2024
About Me
3/20
2024
IREI Infrastructure
OBJECTIVE
Develop a powerful infrastructure to support and service research in Iceland
Which include:
IT consultancy,
access to powerful HPC,
solutions for data storage
and data sharing
GRANTS
Infrastructure fund has granted 455 m.kr to the project so far
Foundation for the infrastructure is up and running
Further development and implementation underway
USAGE
Over 230 registered users, a significant increase from 80 2 years ago
4 IREI specialists
Knowledgebase and experience growing
Number of users rapidly increasing
4/20
2024
IREI Services
RÁÐGJÖF
Í boði er …
Almenn UT ráðgjöf
Aðstoð við að finna lausnir
Innleiðing lausna
Umsjón með innkaupum
MIÐLUN GAGNA
Örugg miðlun, innan lands sem utan
Umhverfi fyrir samstarf
GAGNAGEYMSLA
Örugg og skilvirk varðvarsla rannsóknargagna
Upplýsingaöryggi og persónuvernd
Hagkvæmni – greind í stjórnun varðveislu
REIKNIAFL
HPC (CPU and GPU)
Gagnageymslur vegan reikniaðgerða
Mikið reikniafl fyrir hraða úrvinnslu rannsóknargagna
CONSULTANCY
We offer:
General IT consultancy
Problem solving support
Implementing solution
Compile specialized software
Oversee purchases
DATA SHARING
Safe sharing, both within and outside of Iceland
Platform for cooperiation (NextCloud)
DATA STORAGE
Safe and efficient storage of reserach data
Certified storage of personal and sensitive data��ISO 27001 certified data management
COMPUTING POWER
HPC (CPU and GPU)
Directly connected to storage which can host large data
State of the art hardware for efficient analysis of data
Computing Power
Data storage
Consultancy
Data Sharing
Icelandic Jupyter Architecture Co-design by Use cases.
Interaction room
CoE RAISE Final Review – WP2 – Morris Riedel
focus input
focus input
smaller inputs
smaller inputs
[1] M. Book et al., Facilitating Collaboration in Machine Learning and High-Performance Computing Projects with an Interaction Room, IEEE E-SCIENCE 2022, https://doi.org/10.1109/eScience55777.2022.00093
Methodology that Dr. Matthias Book is working on
Revolves around splitting the system into four tables in an interaction room
Simplifies the importance of software systems
Interaction Room Remote Sensing
8/20
20XX
In-progress
Foundation Models for Generalist Geospatial Artificial Intelligence
Framework for building basic geographic models to accelerate the development and dissemination of climate and sustainability
Dr. Rocco Sedona
IAS, Jülich Supercomputing Centre (JSC)
Deputy Head of SDL AI and Machine Learning for Remote sensing
Trained for the use of data from satellite imagery.
2024
Interaction Room Lava Loba
9/20
20XX
Dr. Gro Birkefeldt Møller Pedersen
Running MrLavaloba for generating lavaflow models,
Volcanology expert at the Iceland Met Office
Worked as a PostDoc at the University of Iceland
In-progress
2024
Co-Design System Architecture
Motivation
11/20
2024
Jupyter - JSC
12/20
2024
[3] Jupyter. Jupyter Logo. Forschungszentrum Jülich, jupyter.jsc.fz-juelich.de. Accessed 3 Dec. 2024.
Jupyter - JSC Blueprint
20XX
[2] Jens Henrik Göbbert., Interactive HPC with Jupyterlab, EDIH-Workshop, part 3 Part3_JupyterLab_for_HPC/jupyter4hpc_part3_under-the-hood.pdf · main · jupyter4jsc / workshop-2023.10-jupyter4hpc · GitLab
Background UNICORE
14/20
2024
Separates into three different layers
[4] UNICORE. UNICORE Logo. unicore.eu
Background Unity IdM
15/20
2024
Offers a user-friendly way on setting up on managing user identities also compatible with Unicore
Offers many authentication settings, Oauth and SAML
Can be used to setup SSO connection
[5] Unity IDM. Unity IDM Logo. unity-idm.eu, www.unity-idm.eu.
UNICORE/X
Elja Specific Configuration
UTS Server
Landscape
UNICORE �Gateway Server
Elja Specific Configuration
UNITY IDM
UNICORE�Configuration
PORT XYZ
PORT XYZ
JUpy
Jypyter – JSC
Website
Internet
Elja LDAP�Server (Linux)�(no common user database)
UTS Server
2 landscape
LDAP information?
Add users, remove users for what is LDAP used
HI / UTS Configuration
LDAP
HI / UTS Configuration
firewall
UTS sever landscape
PORT XYZ
Software infrastructure Elja
Elja RMS
Slurm
Login node
UNICORE Target �System Interface (TSI)�(run as uid/setpriv)
Elja Specific Configuration�(usually runs as root)
Python�LavaLoba
Pytorch & TensorFlow�
PORT XYZ
Rocky Linux
Hardware infrastructure
EU �HPC �Systems
Elja
CPU
GPU
NetApp
Storage
Discs
Kubernetes Cluster
Plan of Implementation
Plan of Implementation
2. Setup connection to UNICORE/X from personal computer with TSI layer on Login Node
3. Set up connection from Unicore/X server to the TSI
4. Set up Unity IdM and connect to UNICORE/X
5. Set up connecting with Jupyter-JSC
6. Validate setup with Co-design use cases
18/20
2024
= To-do
= In-Progress
= Done
References
[1] M. Book et al., Facilitating Collaboration in Machine Learning and High-Performance Computing Projects with an Interaction Room, IEEE E-SCIENCE 2022, https://doi.org/10.1109/eScience55777.2022.00093
[2] Jens Henrik Göbbert., Interactive HPC with Jupyterlab, EDIH-Workshop, part 3 Part3_JupyterLab_for_HPC/jupyter4hpc_part3_under-the-hood.pdf · main · jupyter4jsc / workshop-2023.10-jupyter4hpc · GitLab
[3] Jupyter. Jupyter Logo. Forschungszentrum Jülich, jupyter.jsc.fz-juelich.de. Accessed 3 Dec. 2024.
[4] UNICORE. UNICORE Logo. unicore.eu, https://www.unicore.eu/ . Accessed 3.Dec. 2024
[5] Unity IDM. Unity IDM Logo. unity-idm.eu, www.unity-idm.eu. Accessed 3 Dec. 2024.
19/20
2024
Takk fyrir mig
Sigurður Detlef Jónsson