Handbook on EOSC Infrastructure Integration

EOSC-Synergy project

Viet Tran (IISAS) on behalf of ESOC-Synergy WP2 team

1

www.eosc-synergy.eu

www.eosc-synergy.eu

Motivations

Starting points:

• 10 thematics services that need infrastructure resource
• ESOC architecture with definitions of the core services
• EOSC marketplace large choice (320+) of different solutions and tools
• Teams of experts

Challenges:

How to bring users and infrastructures together - in a scalable way that avoids vertical solution

2

www.eosc-synergy.eu

Target audiences

The handbook will provide:

• How to integrate computing and storage infrastructure
• Services and tools supported on the infrastructure
• How to integrate thematic services to the infrastructure

​

Target audiences:

• Management of Computer Centres
• Users of the cloud
• System Administrators

3

www.eosc-synergy.eu

Thematic services

• There are 10 thematic services in EOSC-Synergy project with different maturities, requirements and used technologies:
• Most notable challenges:
• Increasing computing resources and performances
• Unified AAI
• FAIR
• Used technologies are very various
• More details about thematic services are provided in the talk “Thematic Service integration in EGI Fedcloud” - Miguel Caballer (UPV) in this session

​

4

www.eosc-synergy.eu

EOSC architecture

• AAI
• Accounting
• Monitoring
• Service and resource catalogue
• Marketplace
• Order management

• Onboarding of new providers and services
• Coordinating resource access requests
• User support coordination

5

www.eosc-synergy.eu

EOSC Core for EOSC Exchange

6

Resource onboarding workflow

Provider

Service

AAI

Proxy

EOSC AAI Federation

Helpdesk

Service Delivery

Research Product Catalogue

Service catalogue

Resource Catalogue

(EOSC Exchange)

Monitoring for EOSC Core Services

Monitoring for Exchange

Helpdesk for Core and Exchange

Order management

Accounting for Services

Accounting for Research Products

Service Provider Dashboard

Res. Products Provider Dashboard

Service accounting

www.eosc-synergy.eu

EOSC Core for EOSC Exchange

www.eosc-synergy.eu

EOSC core services

8

What?

• All available to integrate with different levels depending on the integration scenario
• Support from the technical teams

Why?

• Joining the open science initiative
• Support for operational part of service delivery (SMS)
• New user base (young researchers, long tail of science, interdisciplinary researchers

www.eosc-synergy.eu

EOSC Monitoring

9

5 supported use cases:

​

1. Monitor an Onboarded Service (central one)

service onboarded via the Providers Portal is exposed in EOSC Monitoring WebUI

• Monitor an Infrastructure (community)

supporting custom monitoring requirements (topology,, probes, metrics, etc.)

• Integrate External Monitoring service

accepting data from external sources

• Combine Results of existing ARGO Tenants

topology and the results of multiple tenants need to be combined in a number of reports

• Third-party services exploiting EOSC Monitoring data

the results of the EOSC Monitoring Service in an external service/dashboard

www.eosc-synergy.eu

EOSC Accounting (research products)

10

​

A PUSH Workflow which allows server side real-time tracking using platform specific tracking software or using a generic log file parser based on Python that parses log files. Usage events are dispatched to Matomo Analytics platform by exploiting the platform’s API.The PUSH workflow supports anonymization of IPs.

A PULL Workflow that collects COUNTER CoP usage statistics reports.

How?

• registration of the Provider via EOSC provider’s portal or via OpenAIRE Provide.
• installation of the tracking code
• tracking of usage events from the provider
• validation of tracking of usage events from the provider
• retrieval of usage statistics reports presented in EOSC provider’s portal or at OpenAIRE Provide, or via a SUSHI-Lite API endpoint.

​

www.eosc-synergy.eu

EOSC Resource Catalogue

11

• providers, to register to the EOSC to become eligible to the onboarding of resources
• providers, to onboard their services/research products into the EOSC Service Data
• providers, to view the list of services registered in the EOSC portal and perform a variety of actions such as activate, deactivate, view usage statistics,
• EOSC Portal Onboarding Team (EPOT) members, to manage the onboarding process (approve, reject an application), manage the catalogue of providers and services and audit the validity of the catalogue entries.
• providers of catalogues, to add entire regional or thematic catalogs to the EOSC ecosystem

www.eosc-synergy.eu

EOSC HelpDesk

3 levels of interoperability with helpdesk, which correspond to the three integration paths will be offered:

• Full integration: this path corresponds to the integration of community helpdesks described in the previous section, which implies full synchronisation between EOSC Helpdesk and community helpdesk. This integration can be achieved by application of a set of helpdesk REST APIs. The exact integration guidelines should be defined based on the specifications agreed with the community.
• Ticket redirection: in this integration the EOSC helpdesk isto be used only as a contact point to redirect the initial request to the provider’s or community mailing list without further integration.
• Direct usage: in this integration the EOSC helpdesk can be used as the ticketing system for the community and their onboarded services.

12

www.eosc-synergy.eu

EOSC order management

13

www.eosc-synergy.eu

Joining EOSC

Why joining EOSC

• Extended scalability, beyond the size of one local cloud
• Extended availability for critical services
• Federated AAI
• A large ecosystem of tools and services for federated EOSC infrastructure.
• Monitoring and accounting
• Efficient communication and user-support
• Professional service management procedures

14

www.eosc-synergy.eu

Resource centers: Joining EOSC

How to join

• Organisational prerequisites:
• Join your national grid initiative (NGI) to obtain an entry in the GOCDB and ensure you can support the relevant policies.
• Technical prerequisites:
• Have OpenStack installed.
• Install the required additional tools and services for monitoring, authentication, accounting, networking, ...
• Allocation policies:
• Make decisions regarding which Virtual Organisations (VOs)

More details https://docs.egi.eu/providers/cloud-compute/

15

www.eosc-synergy.eu

Users: Joining EOSC

• Registration to AAI
• Joining or creating VOs
• Asking providers to support VO

​

More details are available at https://docs.egi.eu/users/aai/check-in/

​

• Access resources via available services and tools (see next slides)

16

www.eosc-synergy.eu

Services and tools

• AAI
• oidc-agent, mytoken, ssh-oidc, …
• Cloud compute
• FedCloud client, Dynamic DNS, Infrastructure Manager, …
• Storage
• OpenStack Cinder/Swift, rclone, EGI DataHub, ..
• HPC
• udocker, ssh-oidc, …
• SQA
• SQAaaS, …
• Training
• Online training platform, …

17

www.eosc-synergy.eu

Pre-release of the Handbook

• The pre-release of the Handbook is available as https://handbook.eosc-synergy.eu/
• Some sections are still not complete, final release is expected at the end of October
• Comments and suggestions for improvements are welcomed

18

www.eosc-synergy.eu

Summary

EOSC-Synergy is bringing users and infrastructures together.

​

The handbook is for different target audiences:

• Resource centres (managers, system admins)
• Users of the cloud (VO managers, app developers, end-users)

Everyone can find what they needs without over-complicated information

19

www.eosc-synergy.eu