NOSC: Nordic Open Science Cloud

NeIC Executive Manager

Michaela Barth <caela@kth.se>

​

2017-10-25

NeIC Internal

Espoo

​

​

Outline

• EOSC
• Initial Steps so far
• Importance and Opportunities
• Redefining the Scope
• Next steps: Stakeholder meeting

​

​

​

SPEAKER | Michaela Barth caela@kth.se

​

EOSC

European Open Science Cloud (EOSC) and EOSC-Hub

• What do we know about EOSC and EOSC-Hub?
• Eoscpilot.eu Pilot project to support the development of the first phase of the EOSC
• EOSC-Hub (EINFRA-12-2017) for setting up the technical infrastructure, from the building blocks provided by EGI, EUDAT and INDIGO-DataCloud, intended to start January 2018
• EOSC Advisory group: Commission High Level expert group (chaired by Silvana Muscella) with 10 experts from June 2017 - June 2018, first report defining the EOSC.
• Additional EOSCINFRA calls pending

​

SPEAKER | Michaela Barth caela@kth.se

EOSC coverage in the Nordic countries

• Slide under construction!
• Eoscpilot.eu Pilot (coord: STFC) project to support the development of the first phase of the EOSC, 33 participants
• 10 science demonstrators, one of them being WLCG!
• Others:
• ICOS Lund (SE) part of ENVRI
• Social Sciences include e.g. CLARIN, DARIAH and Digital Humanities
• Lot of Life Sciences!
• EOSC-Hub (EINFRA-12-2017): The scope of the EOSC-hub project is to create the integration and management system of the future European Open Science Cloud that delivers a catalogue of services, software and data from the EGI Federation, EUDAT CDI, INDIGO-DataCloud and major research e-infrastructures. This integration and management system (the Hub) builds on mature processes, policies and tools from the leading European federated e-Infrastructures to cover the whole life-cycle of services, from planning to delivery. The proposal was submitted at the end of March and successfully reviewd and favourable evaluated by he EC. 24 EUDAT partners were part of it, altogether 74(see proposal! EGI states >100 ??) beneficiaries. The consortium will be led by the EGI Foundation. Currently preparing grant agreement?
• CSC, DeIC, SNIC and UNINETT Sigma2 officially listed as partners in proposal

​

​

SPEAKER | Michaela Barth caela@kth.se

​

Initial Steps so far

Chronological

• Damien Lecarpentier was invited by NeIC’s Pool Competences working group to actively participate in a workshop at the NeIC2017 conference on Pool Competencies and Research Community Engagement http://neic2017.nordforsk.org/workshops/poco/ where he suggested a Nordic Open Science Cloud (slides online).
• Some excited discussion during the conference and on Social Media followed.
• NOSC draft proposal document by Damien was discussed during Provider Forum [https://wiki.neic.no/int/Provider_Forum:meeting/2017-08-23] and recommended to be discussed at the NeIC Board
• With the help of Executive Manager Tomasz Malkiewicz, Damien Lecarpentier wrote a first Project Directive based on proposal document
• September 13th 2017: NeIC endorses the general principles of the EOSC Declaration.
• September 21st: NeIC Board discussed the Project Directive and commented that the scope should be more clearly defined and that an additional management layer was not desirable; the content currently described does not justify the budget proposed, but a commitment towards EOSC is clearly needed and a budget post was already reserved nonetheless. A person can be funded for preparation work, but no PM yet.

SPEAKER | Michaela Barth caela@kth.se

​

Importance and Opportunities

Why a Nordic Open Science Cloud (NOSC) is important

• The desired EOSC concepts are widely playing into what we are already doing within NeIC

“We could rename NeIC to NOSC and we are done.”

• Funding agencies are interested in the EOSC and bound to/will be obliged to fulfill it.
• -> has to be checked for every country!
• NOSC could help us via EOSC to advertise and disseminate all the good things we are already produce and live in our projects.
• A NOSC would clearly go with the NeIC vision to become a global role model for cross-border distributed and sustainable e-infrastructure services as outlined in the NeIC strategy for 2016-2020.
• Getting in closer contact with the funding agencies and other stakeholders and engaging them more is desirable

SPEAKER | Michaela Barth caela@kth.se

NOSC opportunities

• Advertise and leverage current ongoing activities within the Nordics towards EOSC
• Looking closer at the synergies between especially Glenna2, Dellingr, Tryggve2 and Pool Competences (Data Management) and Resource Sharing Focus Areas
• Making participation of the National Funding Agencies in a reference group as a requirement
• Coordination arena
• Doing something specific that benefits the researcher
• Not so big, not so much overhead compared to EOSC
• @NBlomberg“There is nothing like NeIC and NordForsk in Europe. It is unique”
• “If the Nordics can’t do it, Europe can’t do it” (Damien Lecarpentier)
• Advice our ministries in FAIR data principles?
• Syncing Nordic Funding cycles and policies

SPEAKER | Michaela Barth caela@kth.se

​

Redefining the Scope

NeIC Board input

• “The proposal in its current form was vague and it was unclear what the proposed 4FTEs would do. In addition to being a platform bringing together stakeholders and engagement on policy level, the proposal could also look at tangible services that will benefit researchers. The Board was positive to the idea of NOSC, but did not see that the content of the proposal in its current form justified the proposed budget.”
• The Board discussed participation from the funding agencies in NOSC, and agreed that rather than including them as partners they should be part of the NOSC reference group.
• Action point 17-40a: NeIC will organise a meeting for NOSC stakeholders.
• Action point 17-40b: NeIC will provide an updated PDI for NOSC for the December Board meeting.
• Decision 17-40: To hire a project manager for NOSC at the current stage would be premature. However, NeIC can hire temporary personnel to support in developing the project and write the proposal.

SPEAKER | Michaela Barth caela@kth.se

​

Next steps

Scoping of a revised activity or action

• Mapping exercise: map per country and map per project (Glenna2, Dellingr, Tryggve2 and Pool Competences (Data Management) and Resource Sharing Focus Areas) on what could be contributed already, and where synergies are seen

​

• Start with a meeting between Project office (Tomasz and Michaela) and PMs of mentioned projects

​

• Based on that we can start updating the PDI (see Board comments and also direct input from PF members)

SPEAKER | Michaela Barth caela@kth.se

Stakeholder meeting

• Who should be invited?
• SG members of projects?
• Funding agencies (got contactlist)
• NeIPs?
• EOSC (Silvana?)
• Maybe the stakeholder meeting is already the project?
• Outline the potential conflict between:
• What do we expect from EOSC?
• What does EOSC expect from NOSC?

SPEAKER | Michaela Barth caela@kth.se

​

What we have so far

Overview input

• Original PDI by Damien and Tomasz
• Feedback PF members (Tomasz sent NO, SE, FI feedback to Michaela)
• Damien should have more detailed feedback from DK
• NeIC Board minutes with feedback
• Glenna SG statements

SPEAKER | Michaela Barth caela@kth.se

NeIC XT considerations

• No official statements and commitments from the funding agencies yet on NOSC, this is one of the reason why we need a stakeholder meeting with them
• Coordination and synergy effects should be a matter of the XT, not a Project Manager
• Needs understanding of projects and NeIC strategy

​

• No concrete steps/items for support were given when supporting the EOSC
• (has to be covered in the PDI)
• NeIC XT will do a separate discussion on which points in the EOSC declaration could/should be leveraged to our new strategy implementation.
• NeIC doesn’t have a reference group. NeIC could have e.g. bi-annual meetings with NeIPs and funding agencies, thereby providing a coordination arena/ forum also for the research council and NeIC funders
• (Research Infrastructure group @ NordForsk, has no role towards NeIC)

SPEAKER | Michaela Barth caela@kth.se

​

Glenna2 and NOSC

Glenna 2 overview: The four aims

• Supporting national cloud initiatives to sustain affordable IaaS cloud resources through financial support, knowledge-exchange and pooling competency on cloud operations.

​

• Using such national resources to establish an internationally leading collaboration on data-intensive computing in collaboration with user-communities.

​

• Leveraging the pooled competency to take responsibility for assessing future hybrid cloud technology and communicate that to the national initiatives.

​

• Supporting use of resources by pooling national cloud application expert support and create a Nordic support channel for cloud and big data. The mandate is to sustain a coordinated training and dissemination effort, creating training material and providing application-level support to cloud users in all countries

SPEAKER | Michaela Barth caela@kth.se

Glenna2 NOSC interpretation and contribution

• SG decision 9th of June 2017: We can call Glenna2 the Nordic Science Cloud experiment
• SG interpretation of the EOSC/NOSC: “Open as the A in FAIR”
• Contribution by AIM-2 (A collaboration platform for data-intensive computations)
• Target 1: Kubernetes and Nordic Ubernetes setup (12 PMs)
• End user application store with ability to deploy applications across kubernetes instances with authentication/authorisation support from Dataporten (as well as Edugain through Dataporten). This task will also include investigating Red Hats OpenShift environment.
• Spark and Notebook as the applications which above mentioned store will enable users to deploy on the kubernetes clusters across Nordics. Blueprints (Pebbels CSC) framework potential solution for automatic classroom deployments.
• Investigate Deep learning with GPU on kubernetes (Investigative task for 2017 with the aim of concrete results in 2018)
• Deliverables:
• Minimum Viable Product (MVP) during 2017
• demo at NeIC AHM18.
• Report on OpenShift vs. Kubernetes results.
• Achievements so far: Setting up a personal Kubernetes container orchestration platform in OpenStack
• Aim2 achievements show clear benefit for use cases, Kubernetes container orchestration not set up in SE yet, also of interest within Tryggve and will be tested out there

SPEAKER | Michaela Barth caela@kth.se

​

Tryggve2 and NOSC

Tryggve2 NOSC interpretation and contribution

• The aim of coordinating Nordic interests in EOSC is highly supported
• More weight in influencing policies and decisions
• Coordinating the work and avoiding duplicate tasks
• This will bring best benefit for Nordic communities from the European developments
• Tryggve2 major goal is to develop and facilitate access to a top-level e-infrastructure for sensitive data; and this has several connections to NOSC / EOSC topics, including:
• Policies for accessing services across countries
• Policies for handling sensitive data
• State-of-the-art secure processing and data services
• Operating and providing access to data repositories
• Portable software installations (containers)
• Topics for collaboration between proposed NOSC and Tryggve2
• Evaluation and review of legal and ethical issues regarding sensitive data use across borders (Code of conduct, guidelines for researchers, guidelines for service providers, ...)
• Policies for access across countries (aligning user processes, sharing of cost for infrastructure access, ...)
• Integration to common service portfolio, i.e. how can the Nordic secure systems be visible thorugh EOSC
• Collaboration with ELIXIR Competence Center task in the EOSC Hub projects (see next slide)

SPEAKER | Michaela Barth caela@kth.se

ELIXIR Competence center task in EOSC Hub

• Several of the aims for ELIXIR competence center within EOSC hub are of high interest for Nordics, and are aligned with Tryggve2 targets. See task description below.

​

T8.1 ELIXIR (Lead: EMBL-EBI; Participants: MU) [PM1-PM36]

• The CC will demonstrate the analysis of life-science data on EOSC compatible cloud resources by drawing on leading sites of the ELIXIR Compute Platform. The CC will:
• Make the partners’ cloud resources compatible with the EOSC framework. Develop a costing model to inform future virtual access models.
• Design and demonstrate a container data packaging and distribution framework from a primary data provider to a hosting cloud site for data analysis runs.
• Integrate and operate the Reference Data Set Distribution Service to enable the distribution of ELIXIR Core Data resources to cloud sites.
• Operate the life science AAI within the EOSC AAI.
• Engage with T2.5 on Code of Conduct around how EOSC could host and analyse human data.

​

SPEAKER | Michaela Barth caela@kth.se

​

Dellingr and NOSC

Dellingr Possibilities

Dellingr is about HPC resource sharing

​

Legal issues on resource sharing are to be studied in Dellingr… could be interesting for NOSC

• The legal positions of each country to share resources to be determined
• Similar will be needed by NOSC
• High-level track of Dellingr

​

Also the conditions under which resources can be shared by each national provider

• Do resources need to be “balanced”?
• How is this to be done?
• Resource balancing mechanism.
• Low-level track of Dellingr

SPEAKER | Michaela Barth caela@kth.se

​

EISCAT_3D and NOSC

EISCAT_3D

​

EISCAT_3D will require significant resources to perform

• Parallel computing tasks
• Analysis by users

​

Will require cluster computing with easy access (Cloud?)

• EISCAT users need easy AA system

​

Users will come from Nordics and beyond

​

EISCAT_3D is an obvious use-case for a Cloud infrastructure

​

Already using VMs in a limited manner through the EGI CC project (to be EOSC hub CC)

SPEAKER | Michaela Barth caela@kth.se

​

CodeRefinery and NOSC

CodeRefinery requirements

• CodeRefinery needs a long term commitment to a GitLab source code repository hosting:
• IPV6 support
• targeting 1000 projects centered in Nordics but allowing collaborators outside
• support to host GitLab pages (https://about.gitlab.com/features/)
• support to host Docker registry (https://about.gitlab.com/features/)
• ideally built on OpenStack reusing in-house solutions and expertise

​

• Optionally CodeRefinery would like to provide:
• limited continuous integration runners based on GitLab CI to few selected projects

​

• Both services are relatively cheap in terms of CPU, disk, and maintenance (which can be incremental next to existing services).

​

• So far: Beta test GitLab @ CSC, contract negotiations with DeIC for moving there

​

​

​

SPEAKER | Michaela Barth caela@kth.se

​

NT1 and NOSC

NT1 NOSC interpretation and contribution

• WLCG is a science demonstrator within EOSC pilot
• EOSC is seen as a good push for long-term data preservation and reusability of LHC data
• NT1 provides large scale storage services to WLCG
• Bit preservation is an essential part of data preservation
• DMPs etc out of scope for NT1 though, we’re providing infrastructure and data management is a task for the user communities

​

• “The EOSC should incentivise the re-use of existing building blocks, state-of-the-art services and solutions”
• NT1 is running state-of-the-art storage and computing services for WLCG
• WLCG is a fore-runner on large-scale data storage
• WLCG and OSG are the only two distributed computing infrastructures that does production at scale - essential to learn from in order to build something that works

​

• NT1 is involved in several of the EOSC-hub projects

SPEAKER | Michaela Barth caela@kth.se

​

Ratatosk and NOSC

“NeIC’s interest in training is focused on e-Science and e-Infrastructure skills among scientists and e-Infrastructure personnel in the Nordic region, for the purpose of raising the quality of the scientific output.”

Ratatosk Training programme

• Established recently (September 2017), yearly funding 0.5 MNOK
• 2 parts of a Mobility Enhancement Programme:
• “Student” Mobility Travel Grants targetting
• e-infrastructure users (researchers)
• E-infrastructure staff
• Course mobility: Open Call (after pilots)
• Train the trainer
• Open online material encouraged
• Streaming and videos
• EOSC: “Skills” (within Data culture and Fair data) within EOSC declaration: Lifting and updating education in research data management, data stewardship and data science to a a Nordic level
• (not decided yet) Online material collection
• NeIC event management service (indico)
• (not decided yet) streaming/screencasts guideline to support events

SPEAKER | Michaela Barth caela@kth.se

​

NeIC Pool Competencies FA and NOSC

Pool Competencies (NeIC Focus Area 1)

• Working group (PoCo WG) in effect since 2016
• Training chosen as a pilot case:
• Pool Competencies Training Status Overview (Del. 1 of WP1) finished
• Joint calendar and a training metaportal established https://neic.nordforsk.org/training/
• NeIC Training policy decided
• Agreement on add-on questions to national surveys
• Establishing of Ratatosk and becoming SG of Ratatosk
• Data Management working group
• Priority chosen by NeIC Board after suggestions from PoCo WG
• One2One interviews (in progress)
• Data management specialist hired https://neic.no/news/2017/10/23/andreas-j/
• Potential collab interests:
• Common Nordic maDMP tool (using DoI)
• Template for Data Management Policies
• Open Access and Archiving of Research data?

SPEAKER | Michaela Barth caela@kth.se

​

NeIC going FAIR?

Findable, Accessible, Interoperable, Reusable

• Independence:
• NeIPs can tap into this independently, independent of EU funding streams
• driven by countries, not by European Commission (not politically biased)
• endorsed by the Commission as a vital contribution to EOSC
• Already concentrating just on this could cover the funding agencies’ obligations
• wider international reach (strengthening NeIC’s global role model aspirations)

​

• NeIC’s role could be to fund an office function that
• supports implementation of FAIR principles in communities across the Nordics
• facilitates coordination meetings between national FAIR points-of-contact in the Nordic countries (assuming that the countries want to self-represent in FAIR governance rather than having a common Nordic representative)
• helps in identifying area or communities that could be targets for joint Nordic efforts
• FAIR Data Governance: Stakeholder engagement and governance structure ready in place (missing authority to enforce it)

​

SPEAKER | Michaela Barth caela@kth.se