Harmonizing research metrics and indicators across the ecosystem���Dr. Kelly Cobey�Scientist, University of Ottawa Heart Institute�Associate Professor, SEPH, uOttawa�co-chair, DORA �kcobey@ottawaheart.ca

sfdora.org

@DORAssessment on most social media platforms

Declaration On Research Assessment (DORA)

Mission: Advance practical and robust approaches to improve how research is assessed across all scholarly disciplines globally

​

• The declaration demonstrates community support for change
• Currently signed by over 25k individuals and organizations (e.g. academic institutions, funders, journals, publishers, societies)

​

​

What’s wrong with the current system? �Open Science Example

Open science: A movement and set of practices to conduct research more transparently.

​

No consensus on what open science is, but commonly includes:

Open access, open data, open materials, study registration, citizen science, open education, etc.

​

​

Open science is not “science”

Open science supports equity

5

Patient gets life-changing diagnosis thanks to Open Ottawa HospitalScience - The

Jenna Keindel

Is science a right?

Jenna’s story in her words

Open Science Supports Innovation

• Development of Pfizer vaccine began the same day the genome sequence for SARS-CoV-2 was publicly released

Why isn’t open science the norm?

• “…the concept of Open Science finds a lot of support in theory, yet struggles in practice” (Open Science: Many Good Resolutions, Very Few Incentives)

​

​

• Traditional incentives value the number of publications, journal impact factor, and citations

​

• We need to stop valuing quantity over quality – open science is a quality metric

It is not just open science…

• Patient/Citizen engagement
• Indigenous engagement
• Knowledge mobilization
• Reproducible and rigorous methods
• Teaching and service
• Equity, diversity, and inclusion

DORA Guidance on quantitative indicators

• Recognized tendency to rely on quantitative indicators

​

• Quantitative metrics are inherently reductive

​

• Explains issues with metrics beyond JIF
• citations, h-index, field-normalized citations, Altmetrics

Citations

• “The citation count of an article is defined as the number of times it is included in the reference list of other articles or books

​

• Citation performance is a lagging indicator that takes time, often years, to turn into a robust signal. It is therefore not well suited to evaluate recent scholarship or to compare researchers at different career stages, or in different disciplines.

​

• Bibliometricians acknowledge that citations reflect the influence of a research article, but this can differ in important ways from what evaluators may really want to determine: the quality and significance of research findings. Citation patterns can be skewed by author and journal reputations; e.g., author status can lead to citation bias, with prominent researchers attracting more citations for similar work than less well-known researchers, a phenomenon long known as the Matthew effect.

​

• Differences in citation patterns that disfavor women are also well documented and should be taken into account when considering citations for researcher assessment. Moreover, since citation data do not indicate whether articles are cited for positive or negative reasons, they cannot be used to indicate research quality without additional supporting information.”

DORA Guidance on quantitative indicators

• Set of 5 principles

​

• What’s the rationale for the indicator?
• Is your approach transparent? (i.e., open data, algorithms, community informed)
• Do aggregate measures make sense?
• How do you consider the reductive nature of the indicator?
• What considerations are there for biases in using the indicator?

Quantitative metrics for monitoring are important.

Project Tool to Advance Research Assessment (TARA)

Three-part research program funded by Arcadia:

1) Survey of US-based institutions to understand institutional attitudes and approaches to research assessment

2) Toolkit of resources to support academic institutions

3) Reformscape: an open database of criteria and standards for hiring, review, promotion, and tenure.

• a project to facilitate the development of new policies and practices for academic career assessment

​

Next Steps

• Tool development:
• January meeting – Developing and implementation guide for academic institutions to support research assessment reform

​

• Coalition building:
• Working closer with international groups to coalesce efforts – joining forces to have more power

​

• Creating space for dialogue
• Provide a forum for exchange on a global scale

​

Thank you! �Dr. Kelly Cobey�Scientist, University of Ottawa Heart Institute�Associate Professor, SEPH, uOttawa�co-chair, DORA �kcobey@ottawaheart.ca

sfdora.org

@DORAssessment on most social media platforms

Institutional change

U. Texas Southwestern Medical Center (USA)

Candidates receive questions before Skype interviews, so they have time to reflect

The goal is to identify thoughtful individuals, not only candidates who process information quickly

https://sfdora.org/2018/05/15/how-to-strengthen-hiring-practices-at-academic-institutions-an-interview-with-dr-sandra-schmid/

Institutional change

Charité University Hospital (Germany) application includes:

• Contribution to research field
• Open Science
• Team science
• Interactions with stakeholders

Staff member sits on hiring meetings as a neutral party to promote balanced discussions

https://twitter.com/dirnagl/status/970227847943114752