Three perspectives on
Open Science �in research assessment
/010203
Purpose, Context and Scope
In all these developments, Open Science is mentioned explicitly to become part of new assessment procedures. �
What does this mean in practice? �
In this slide deck, we provide a digest of current developments, where we focus on the role Open Science will play in:�
The recognition and reward structures at Dutch universities are being reevaluated. Developments that are spearheading this transition are:�
All content in this slidedeck is licensed under a:
NPOS - Accelerate
Open Science
Three Perspectives
01/ Evaluation of Researchers
02/ Evaluation of Research Proposals
03/ Evaluation of Research
THREE PERSPECTIVES ON OPEN SCIENCE �IN RESEARCH ASSESSMENT
Evaluation of
Researchers
/01
/010203
THREE PERSPECTIVES ON OPEN SCIENCE IN RESEARCH ASSESSMENT
“Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist’s contributions, or in hiring, promotion, or funding decisions.”
DORA, intends to halt the practice of correlating the journal impact factor to the merits of a specific scientist's contributions. According to this statement, this practice creates biases and inaccuracies when appraising scientific research. Recommended actions are:�
DORA AND ITS IMPLICATIONS FOR ASSESSMENT OF RESEARCHERS
/010203
Implications for Open Science
Choosing an Open Access journal as an outlet for your research must not influence the evaluation of your work, even when that journal has a lower impact factor.
Other research outputs, such as datasets, materials or software are also considered during evaluation.
Use a range of article metrics and indicators (including personal/supporting statements) as evidence of the impact of individual published articles and other research outputs
“Open science is bound up inextricably with the modernisation of the system of recognition and rewards. It requires time and attention from academics that cannot be automatically translated as traditional academic output such as publications, but which can have a significant impact on society, science and academia”
This calls for a new system of recognition and rewards of academics and research that:
POSITION PAPER ON SUPPORTING OPEN SCIENCE � ASSESSMENT OF RESEARCHERS
/010203
Implications for Open Science
Each institution appoints a committee to create support for this new system of Recognition & Rewards and translate it into institution-specific assessment criteria and narratives for all key areas and team achievements.
Each institution develops courses on academic leadership aimed at the university’s key areas: education, research, impact and (in university medical centres) patient care.
Evaluation of
Research proposals
/02
/010203
THREE PERSPECTIVES ON OPEN SCIENCE IN RESEARCH ASSESSMENT
POSITION PAPER AND ITS IMPLICATIONS FOR � ASSESSMENT OF RESEARCH PROPOSALS
/010203
“Research should be assessed for content and quality, not just for quantity or for the journal it was published in”
Commitment from Dutch research funders NWO/ZonMw to
Focusing on whether the expertise of the researcher(s) involved is/are appropriate for the formulated research question, instead of looking at individual quality
Giving the inclusion of research output on curricula vitae and application forms a more narrative character (e.g. a ‘top 10’ of impactful outputs rather than a complete list)
Relinquishing bibliometric publication indicators (h-index, journal impact factor)
Create an array of funding instruments with clearly differentiating criteria to take account of a more diverse group of researchers
Provide training and instruction to assessment committees, and structuring them in a way that recognizes and embraces diversity in research
Place more emphasis on team science and on cross- disciplinary collaboration in assessment of research proposals
Evaluation of
Research
/03
/010203
THREE PERSPECTIVES ON OPEN SCIENCE IN RESEARCH ASSESSMENT
“The evaluation committee considers whether the research unit reuses data where possible; how it stores the research data according to the FAIR principles; how it makes its research data, methods and materials available; and when publications are available through open access”
In 2021, the new Strategy Evaluation Protocol (SEP) will be implemented. The goal of this protocol is to evaluate a research unit in light of its own aims and strategy. An assessment committee of independent experts assesses the performance of the unit based on the self-evaluation and a site visit. Based on the self-evaluation and the site visit, the assessment committee assesses the performance of the unit. Three main assessment criteria will be important:
1) research quality, 2) societal relevance and 3) viability.
In the SEP, Open Science plays an important role in how research quality and societal relevance can be achieved.
In the assessment, the research unit reflects on how it involves stakeholders, to which extent the research unit opens up its work to other researchers and societal stakeholders, how it pays attention to other aspects of open science and what its future plans are in this respect.
SEP AND ITS IMPLICATIONS FOR � ASSESSMENT OF RESEARCH
/010203
Implications for Open Science
Room for Everyone’s talent
In 2019, a coalition of Dutch knowledge institutes (VSNU, NFU, KNAW) and research funders (NWO, ZonMW) published a position paper that lays out plans to reform recognition and reward structures.
The reform resolves around five pillars:
Resources:
Strategy Evaluation Protocol
The main goal of the Strategy Evaluation Protocol (SEP) is to maintain and improve the quality, societal relevance and viability of research.
The SEP 2021-2027 states that four specific aspects must be
addressed in research evaluation:
1) Open Science
2) PhD Policy and Training
3) Academic Culture
4) Human Resources Policy
San Francisco Declaration on
Research Assessment
The Declaration on Research Assessment (DORA) was developed in 2012, and has become a worldwide initiative that recognizes the need to improve the ways in which the outputs of scholarly research are evaluated.
Recommendations include:
More info
This slide deck is part of the outputs of the Accelerate Open Science project, a project within the scope of the National Programme Open Science. Aim of this project is to foster initiatives that promote Open Science in the Netherlands.
This is done, for instance, by sharing good practices for researchers undertaking Open Science and clear guidance on topics such as Open Access, FAIR data, Recognition & Rewards and Plan S. The project connects and fosters local initiatives, such as the new Open Science Communities at different universities. Moreover, it organises the first Netherlands National Open Science Festival, which provides researchers the opportunity to learn about the benefits of various Open Science practices and to meet peers that are already working openly.
All content in this slidedeck is licensed under a:
NPOS - Accelerate
Open Science