1 of 29

Traditional metrics, altmetrics and researcher profiles: A survey of faculty perceptions and use.

Anne Ferrier-Watson

Academic Liaison Librarian, University of Waikato

LIANZA October 2019

annefw@waikato.ac.nz

2 of 29

What’s the connection?

Research impact is the holy grail of academia.
Traditional metrics and alternative metrics indicate the academic and societal impact of research.
Researcher Profiles help make research more visible.
The more visible a researcher’s work, the more chance it has to generate impact.

3 of 29

Research Impact

…the traceable influence that a scholarly entity has on other research in the discipline

…the provable effects of research in the real world… the changes we can see (demonstrate, measure, capture) beyond academia (in society, economy, environment) which happen because of our research

Roemer, R. C., & Borchardt, R. (2015). Meaningful metrics: A 21st-century librarian’s guide to bibliometrics, altmetrics and research impact. Chicago,IL: ACRL, p.236.

Bayley, J. & Phipps, D. (2017). Real impact: Impact literacy workbook. Retrieved from https://www.emeraldpublishing.com/resources/impact-literacy-workbook, p.3.

4 of 29

Metrics

Traditional

Alternative

5 of 29

6 of 29

Researcher Profiles

7 of 29

Why did I do this research?

I began in my current role in late 2015, during the lead up to the 2018 PBRF round, all these terms were new to me, I had no idea how they inter-related, yet I was expected to ‘get out there’ and assist academics. to connect their profiles – ORCID, Scopus, Researcher ID, Google Scholar, and to identify metrics for their publications.

I’m the Arts & Humanities Librarian and most academics in my areas didn’t engage with me about metrics or profiles.

So.. on one hand I’m hearing from my colleagues and from the wider institution that we need to help academics link their profiles, claim their publications, and grab hold of their metrics, so they can tell the best story in their PBRF portfolios.

On the other, many of those I engaged with in Arts didn’t know or didn’t care about them, or, if they did, their research outputs were poorly represented in the tools used to provide metrics.

8 of 29

What did I do & who responded?

9 of 29

I asked about

Type of research outputs produced
Sources of metrics
Understandings of metrics
Metrics as a reflection of value of scholarly work
Role of metrics in academic promotion and research assessment – current and ideal
Researcher profiles – motivations, preferences, concerns

10 of 29

Research Outputs

Journal articles

Conference papers

Book chapters

Books

11 of 29

Sources of Traditional Metrics

12 of 29

Sources of Alternative Metrics

13 of 29

What did I learn?

Disciplinary alignments
Understandings
Accuracy
Academic promotion & research assessment
Evaluation gap

Wouters, P. (2017). Bridging the evaluation gap. Engaging Science, Technology and Society, 3, 108-118.

1. As other research has shown, there are clear disciplinary alignments with these measures.

2. Sciences consider traditional metrics do somewhat reflect the value of their work, and they are important to promotion and research assessment. Consequently, they report better understanding and use.

3. Arts & Humanities are the opposite.

4. Social Sciences are somewhere in the middle.

Across the board people reported higher understandings and perceptions of accuracy of traditional metrics compared to altmetrics.

Significantly, no-one considered traditional metrics extremely accurate.

Around a quarter in each discipline did not know how accurate metrics are.�

The main objection to traditional metrics was the “evaluation gap” (Wouters, 2017) where traditional metrics are not considered a relevant measure of the work produced or the audience it benefitted.

This perception of (in)accuracy provides some indication as to why respondents do or do not use metrics.

Even though none believe they are highly accurate they still use them. This goes to the heart of the engagement question, and is linked to the mechanisms of the institution that perpetuates the will to measure.

14 of 29

Themes

(Ir)relevance
Journal ranking hegemony
Inequities
Manipulability
A way of measuring
Altmetrics add value
Institutional imperatives

15 of 29

(Ir)relevance

Altmetrics reflect the publicity given to research rather than the quality of the research.

Arts & Humanities researcher

“Impact factors” appear to be about being read and cited. This has nothing to do with actual impact in the field of education i.e. practices in classrooms.

Social Sciences researcher

16 of 29

A way of measuring

It’s quantitative.

Sciences researcher

Some kind of assessment of research is important to reward and provide further opportunities for those doing important research and this is, all things considered, probably the best of the quick methods to do so.

Social Sciences researcher

Before PBRF, some women academics were marginalised in their scholarly fields (in NZ) until PBRF showed that they are internationally recognised, followed, cited and celebrated.

Social Sciences researcher

17 of 29

Journal ranking hegemony

I always look at Scimago before deciding on a journal to which to submit a manuscript. I only submit to Q1 journals.

Social Science researcher

Value of research output must consider knowledge contribution to your research area and not the journal it is published in.

Sciences researcher

18 of 29

The inequities of metrics

High-status Northern hemisphere academic institutions appear to be the baseline for a lot of metrics, thus putting NZ researchers at a disadvantage due to geography and levels of institutional support.

Arts & Humanities researcher

If you only worked 3 of the last 5 years this will affect your H-5 index but is not likely to be taken into account by anyone comparing your H5 with any one else’s.

Science researcher

19 of 29

Manipulation

It's a game that can be 'rigged' / played well or poorly, resulting in more or fewer citations or altmetrics purely on the basis of marketing / strategic work rather than the actual quality of your work.

Sciences researcher

20 of 29

Altmetrics add value

I believe that the non-traditional and the traditional metrics can work well together to provide a clearer picture of the output.

Social Sciences researcher

Altmetrics appear to be more focused on the contribution of knowledge to the specific research environment as opposed to journal rankings.

Sciences researcher

21 of 29

Institutional imperatives

I had 6 years doing sponsored research. Metrics do NOT capture end user impact. With the continued push towards metrics one impedes one's career by engaging with end users.

Sciences researcher

They become a stick to beat us with- i.e. a 'reason' why promotion is turned down.

Social Sciences researcher

Making decisions about journals to target (I hate the system but at the same time have to function within it - while looking for opportunities such as this survey to highlight issues).

Social Sciences researcher

22 of 29

Researcher Profiles

23 of 29

Researcher Profiles

85

77%

Held61%

85%

77%

66%

61%

Preferred

24 of 29

Which profiles best meet your needs

A comprehensive repository, easy to upload, share, and create projects people can follow. Gets the most traffic, used by others in same area, provides relevant notifications about other work.

Comprehensive list of publications generated automatically; citations; visibility; access.

Comprehensive, updated, used to introduce at conferences, leads to supervision requests, connects researcher with the institution; links to ResearchGate and ORCID profiles

Publication list generated by exporting items from other systems, linked to the University, necessary for publication in certain journals.

25 of 29

What profile information would be most helpful?

26 of 29

Profiles allow me to:

27 of 29

What was that connection again?

Research impact is the holy grail of academia.
Traditional and alternative metrics can help to indicate the academic and societal impact.
Profiles help make research more visible.
The more visible a researcher’s work, the more chance it has to generate impact.

28 of 29

Implications for librarians

It’s complicated. There’s no magic bullet.
Ask our academics

What does research impact mean to you?
Do you care about the impact of your research?
How do you know your research has had an impact?
What tools help you

a) generate impact
b) gather evidence of impact

Pimp my profile

Before “know the landscape”

It’s complicated, and there’s no magic bullet. Metrics in particular is a very vexed area, and my research has highlighted ambivalences about their relevance and about institutional factors that maintain their currency. Contrasting perceptions are evident, and context dependent.

Engagement with researcher profiles is more straightforward. They increase visibility and collaboration, and to a lesser degree increase citations and metrics. Time is a barrier, and there is a general need to know how to better harness them.

My research was driven by a desire to know the landscape of the people we serve. By the end, I realised I had been asking the wrong questions. I should have been asking THESE questions which are predicated on the assumption that researchers actually care about impact (and they should), and puts the researcher at the centre.

Read questions

Equipped with a deeper understanding of an individual researcher’s position, and a discipline specific awareness of the impact arena, we can tailor individualised support, instead of blithely promoting the tools of impact measurement and research promotion.

We are still in ‘figuring this out’ and our approach needs to be more holistic and collaborative, rather than working one to one in isolation.

Thompson and French provide an excellent model of collaboration, which lead to improved outcomes for the faculty, which, after all, is our reason for being!

Thompson, E, E. & French, S. (2016). Pimp my Profile and the researcher profile health check: Practical, individualised researcher support initiatives co-created by library and faculty. ALIA National Conference 2016. Adelaide. Retrieved from https://eprints.qut.edu.au/98649/

29 of 29

Traditional metrics, altmetrics and researcher profiles: A survey of faculty perceptions and use.

Anne Ferrier-Watson

Academic Liaison Librarian, University of Waikato

LIANZA October 2019

annefw@waikato.ac.nz