1 of 25

Welcome to this webinar on Impact @ ELIXIR

This session will be recorded

Please remain muted unless you are invited to speak by the Chair

This meeting will be run in line with the ELIXIR Code of Conduct

Please use the “hand-raising function” to indicate you would like to contribute directly

Please use “Chat” for further comments or discussions

The meeting will start shortly

.

Slides and video recording to be made available at: https://elixir-europe.org/events/webinar-impact-evaluation

2 of 25

Increasing capacity in impact evaluation in a life science research infrastructure

Kim De Ruyck (ELIXIR Belgium), Ana Melo (formerly ELIXIR Portugal)�Christine Stansberg (formerly ELIXIR Norway), Erika Balsyte (ELIXIR Hub) Corinne Martin (ELIXIR Hub)

ELIXIR Webinar Series, Tuesday 03 October 2023, 12-1pm CEST

@ELIXIREurope

/company/elixir-europe

www.elixir-europe.org

3 of 25

We build bioinformatics (human) capacity and infrastructure in Europe, connect and develop a network of experts, and make 100s of bioinformatics services and guidelines freely available to all, across the globe

Databases

Training

Software tools

Data standards

Compute resources

Scientific & technical experts

ELIXIR – what do we do?

4 of 25

The many stakeholders of research infrastructures

Stakeholders	Main interest(s)
National authorities	Justify large financial investment to e.g. Ministry, Parliament; demonstrate, maintain or develop leadership and attractiveness at national/international levels; demonstrate scientific and economic returns on public investment
Regional/local authorities	Justify investment; increased attractiveness of the area (incl. attractiveness and quality of local higher education institutions); benefits to local businesses and the development of local/regional innovation ecosystems
Research infrastructure funders	Value for money; maximise return on investment. Top priority is usually the scientific and technological impact, but funders often require RIs to demonstrate additional benefits
Research infrastructure initiators	Ex ante (i.e. predicted) assessment to demonstrate potential impact for funders, national authorities and local authorities and help raise funding
Research infrastructure management	Monitor impact on a regular basis to improve performance, and gather information to make the case to funders for sustainability including upgrades
Research infrastructure hosts	Demonstrate the value of the research infrastructure in terms of scientific attractiveness and skills
Scientific community	Advocate for new or updated research infrastructure to foster new scientific knowledge
Civil society/general public	Value for money, new scientific knowledge, general benefit to society (e.g. health, energy, food security). On case by case basis, impact on the environment and/or on local populations

(adapted from OECD, 2019,

https://doi.org/10.1787/3ffee43b-en

5 of 25

ELIXIR’s strategic approach to impact evaluation

Examples?

👇

6 of 25

Instruments to build capacity in impact evaluation

Gaining new knowledge and skills through training and knowledge exchange

Undertaking performance and impact evaluation at Node-level

Dissemination of case study results, good practices and lessons-learned

🟢 July 2021 Led by ELIXIR Belgium, Norway & Portugal 🏁 June 2023

EU-funding: RI-PATHS, ELIXIR-CONVERGE, PathOS, RI-Train projects
ELIXIR’s core funding: a staff exchange project (3 ELIXIR Nodes, + ELIXIR travel grants) & a strategic implementation study (>16 ELIXIR Nodes)

Impact evaluation at Node-level — getting it done

👇

Today’s webinar included!

7 of 25

What’s in ELIXIR’s Impact Toolkit?

https://f1000research.com/documents/12-1127

Corinne

today’s webinar will take you through a selected parts of the Toolkit…

https://f1000research.com/documents/12-1127

This Toolkit collates a range of impact-related materials, including training documents and materials, worked examples, factsheets, lists of indicators, key publications, case studies, and presentation slides. These have been created and/or brought together using a bottom-up approach that tries to reflect the varied needs of Nodes in ELIXIR member countries. The primary audiences of this Toolkit are those in ELIXIR (e.g. operators, developers and managers of bioinformatics resources) wishing to demonstrate and communicate the performance, impact, and ultimately public value of their research infrastructure, to their particular set of funders and stakeholders. Sibling research infrastructures in the life sciences and beyond may also find the Toolkit useful.

https://www.freewordcloudgenerator.com/generatewordcloud (words used)

8 of 25

Intended impacts and pathways to reach them

9 of 25

Research efficiency: we make infrastructure, bioinformatics resources and processes faster, easier to use, and more integrated

Bioinformatics resource uptake:

we work to increase resource usage and appreciation by users

Research infrastructure sustainability:

we work to increase ELIXIR’s visibility with, and appreciation by, its funders

Skills development:

(i.e. human capital), we upskill resource users and service providers

Equal opportunity:

we raise awareness of diversity and inclusiveness

Policy influence:

we shape policy in the area of Open Science and FAIR

Public awareness: we raise awareness of socio-economic and societal benefits of bioinformatics and Open Science

Benefits derived from working together: (i.e. relationship/social capital), we facilitate knowledge-sharing and cooperation

Scientific legacy: we create and disseminate new knowledge on research infrastructure, bioinformatics resources and related guidelines

Adapted from Martin et al. (2021) https://doi.org/10.1111/apce.12328

Main categories of direct impact for work funded by, and through, ELIXIR

Kim

A good place to start on the long journey towards demonstrating impact is to define high-level/broad impact areas that relates to activities you are undertaking

these broad impact categories can be further refined as you progress, but they remain a good place to start
and of course, these impacts feed into even broader impacts such as the UN Sustainable Development Goals

All activities carried out under the umbrella of ELIXIR, i.e. ELIXIR related activities, can be attached to one or more of the impact categories (or areas)

These categories are a useful way to describe what ELIXIR related activities are meant to achieve in terms of impact

[Direct link to slide version] + Navigate to that slide from https://elixir-europe.org/about-us/impact/more-examples]

*********

Main categories of direct impact for work funded by and through ELIXIR (adapted from Fig. 1 in Martin et al., 2021):

Research efficiency: we make infrastructure, resources and processes faster, easier to use, and more integrated.
Scientific legacy: we create and disseminate new knowledge on research infrastructure, bioinformatics resources and related guidelines
Benefits derived from working together: (i.e. relationship/social capital, or the value derived from positive connections between people), we facilitate knowledge-sharing and cooperation
Public awareness: we raise awareness of socio-economic and societal benefits of bioinformatics and Open Science
Policy influence: we shape policy in the area of Open Science and FAIR
Equal opportunity: we raise awareness of diversity and inclusiveness
Skills development: (i.e. human capital), we upskill resource users and service providers
Research infrastructure sustainability: we work to increase ELIXIR’s visibility with, and appreciation by, its funders
Bioinformatics resource uptake: we work to increase resource usage and appreciation by users

Citation:

Martin CS, Repo S, Arenas Márquez J, Blomberg N, Lauer KB, Pérez Sitjà X, Velek P, Melo AMP, Stansberg C, De Leo F, Griniece E, Rothe H, Petryszak R, Smith A, 2021. Demonstrating public value to funders and other stakeholders—the journey of ELIXIR, a virtual and distributed research infrastructure for life science data. Figure 1: Main categories of direct impact for work funded by and through ELIXIR. In: Ann Public Coop Econ. 1-14 (part of Special Issue “Knowledge creation: New frontiers for public investment). URL: https://doi.org/10.1111/apce.12328

10 of 25

Building impact pathway logics

https://doi.org/10.5281/zenodo.3950043

11 of 25

Pathway logics can be as simple or elaborate as the context justifies it

Access a range of worked examples in our collection*

* we decline any responsibility

https://doi.org/10.5281/zenodo.7975300

12 of 25

Experience-sharing on defining impact areas and building impact pathway logics

Christine

building on the theory covered in the previous slides, here is an opportunity to discuss what it looks like in practice (in real life!)

Suggestions:

there is no recipe to follow, impact evaluation is a process to go through, ideally collaboratively
it’s ok for a pathway logic to be messy and to be re-visited iteratively as time goes by (meaning that it can support corrective actions)
it’s ok to keep it simple (activities ➤ outcomes ➤ impacts) and focus energy on what is strategically important
Important to have the right mind-frame (think like an evaluator, do not treat this as a scientific exercise, don’t agonise about causality and attribution)
consider how much ‘control’ one has as a guide between what should be an ‘outcome’ or an ‘impact’ (impacts are where one will have the least control over)
draw.io (free add-on to Google Drive) is a useful software tool
without the support of impact specialists and economists, it is best to focus on how activities contribute to outcomes and impacts, rather than struggling with measuring/documenting attribution and causality in the pathway.
one can start working out pathways from Activities (L to R), or from Impacts (R to L).

13 of 25

Indicators (and other evidence of performance and impact)

14 of 25

Selected efforts to compile indicators relevant to research infrastructures

Source	Description
ELIXIR	Indicators (and other evidence) for monitoring the performance and impact of ELIXIR: initial list in Martin et al. (2021), updated version by Balsyte et al. (2023)
European Strategy Forum for Research Infrastructures (ESFRI)	Key performance indicators (Table 1, p. 9, in Working Group on Monitoring, 2019) mapped against Objectives of greatest relevance for research infrastructures, complemented by ‘data sheets’ (Annex 5) and suggestions for qualitative indicators (Annex 4)
Organisation for Economic Cooperation and Development (OECD)	Indicators (Table 4, p. 22-27, in OECD, 2019) for assessing the scientific and socio-economic impact of research infrastructures
RI Impact Pathways (RI-PATHS) project	Indicators (long list, for inspiration) for assessing the socio-economic impact of research infrastructures, classified as activity, outcome and impact indicators
Open Science Impact Pathways (PathOS) project	Open Science Indicator Handbook, covering various aspects around Open Science itself, their academic, societal and economic impacts, and reproducibility

When time and resources are limited:

15 of 25

Experience-sharing on indicators (and other evidence of performance and impact)

Christine

building on the theory covered in the previous slides, here is another opportunity to discuss what it looks like in practice (in real life!)

Suggestions:

without the support of impact specialists and economists, indicators (and related evidence) are likely to be qualitative towards the ‘impact side’ of the pathway logic

Impacts are typically more difficult to document/monitor/measure/quantify than outcomes and activities, which are located in the spheres of influence and control

Intangible impacts may be best captured by impact “stories”, “narratives” and “testimonials”.
In a distributed infrastructure, it is likely that relevant data are already being collected, meaning that efforts can focus on coordinating, standardising and collating what already exists
It can take a few attempts before finding indicators for certain parts of an impact pathway. It is not always possible, in fact, as some things are intangible and will remain so. Yet, if it is a strategic area of work, it is worth investing energy through repeated attempts

16 of 25

Tactics to build capacity in impact evaluation

17 of 25

A range of approaches!

Jelena Angelis

EFIS

Impact evaluation

Elina Griniece

EFIS

Impact evaluation

Robert Petryszak

Datasome

Data analytics/vizz.

Formal training workshops
Dedicated support calls to Nodes on their chosen case studies
Show & Tell events, …

18 of 25

title

19 of 25

Experience-sharing on tactics to build capacity in impact evaluation

20 of 25

Other impact-related resources

21 of 25

Annotated bibliography

Collates a wide-range of published works that are relevant to assessing the performance and impact of Research Infrastructures
Structure - six main sections, sub-sections grouped by RIs
Key ELIXIR-relevant publications

are marked ⭐

We welcome suggestions for new entries

⭐ https://doi.org/10.7490/f1000research.1119622.1

22 of 25

Questions from the �participants

23 of 25

Thank you

24 of 25

25 of 25

Christine

Corinne

Kim

Christine

Erika

Ana