User engagement and the processes and impact of educational research

Anne Edwards

University of Oxford

February 2014

Overview

• The problem(s)

​

• The new impact agenda

​

• Knowledge exchange and pathways to impact

​

• The societal responsibilities of social science

​

• A cultural historical / knowledge practices view of participatory research

• The implications

​

​

The Research Unit for Research Utilisation: Sandra Nutley

The public policy context of many OECD countries can be described as financially austere, organisationally complex and characterised by decentralised decision making. While several commentators argue that under these conditions the need for timely, accessible and reliable evidence is becoming ever more important, our existing models and maps for connecting evidence, policy and practice often seem ill-suited to the task.

(Campbell Collaboration Colloquium, 2012)

Activist Research

The intellectual project of understanding the world.

​

The political project of changing it.

​

(Nancy Fraser, 2004)

Forms of Research Use – a long-term concern

• Direct (instrumental)

Knowledge-driven

Problem-solving (or engineering)

• Indirect (conceptual)

Social interaction

Enlightenment (or percolation)

• Symbolic

Political

Tactical

​

(Carol Weiss 1979)

​

Social Impact and Knowledge Exchange �

​

• Impact is the demonstrable contribution that excellent research makes to society and the economy. Impact embraces all the extremely diverse ways in which research-related knowledge and skills benefit individual, organisations and nations...

​

• By working with a wide range of partners in the private, public and third sectors, we have the ability to leverage both the resources and expertise to enable us to fund excellent research which benefits both society and the economy.

​

​

• We acknowledge that knowledge exchange encompasses complex and diverse activities which can deliver impact over varying timescales.

​

Pathways to Impact

• Bids to research councils now need to explain impact plans or pathways to impact
• Dissemination is a minor element
• Knowledge exchange is central
• There are implications for research design

A New Impact Industry

​

​

​

​

​

​

​

​

ImpactFinder Tool Helps Universities Measure the Impact of Research

​

RAND Europe offers consultancies to universities on how to measure impact – draws very much on current UK work on impact

​

​

In Australia RAND Europe carried out the Excellence in Innovation for Australia Impact Assessment Trial for the network of technology universities (published in 2013)

​

• 12 Universities
• 165 impact case studies
• 7 panels of 75 experts in total of whom 70% were from outside the university sector

​

They found that:

​

• the external experts could assess impact
• Impact could be assessed on significance and reach
• More attention should have been paid to linking the research to the impact in the case studies
• The transactional costs were high

The New Landscape

• Outputs (4 publications per person) 65%

​

• Impact (impact strategy and impact case studies) 20%

​

• Environment (description, research income, student completions) 15%

Impact in the REF

• ..[is] an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia
• Assessed by an impact statement about strategy and approaches to impact and case studies demonstrating impact

Why?

• Show accountability and good research governance to stakeholders?
• Enhance public understanding of the research field?
• Allow the development of more effective research strategies to increase the likelihood of relevant research outcomes?

Nutley’s models of research impact

Stocks or reservoirs of research knowledge

Political and professional environments and wider society

Nutley: Knowledge-to-action (KTA) models

• A bewildering array of models, frameworks and theories (63 models identified by Ward et al. 2009)
• Drawn from wide range of disciplines - very varied levels of evidential support
• Models focused on a spectrum of concerns (Davies et al 2011)

​

​

Critique of KTA based on Randomised Control Trials (the gold standard)

Need to distinguish between what is true and what is relevant

​

For transfer of policy from RCT to other practice settings – need the same causal structures and the same mediating variables

​

Nancy Cartwright and Jeremy Hardie (2013)

Geoff Mulgan’s Challenge

Today a good deal of conceptual innovation is taking place through practice, with relatively few areas in which academics develop theoretical frameworks which others then apply. More often – in cases as diverse as intelligence-led policing or drugs rehabilitation - the theorists are following behind, trying to make sense of what the practitioners are doing.

(Mulgan 2005: 223)

Relational approaches to knowledge production

​

[t]he universities are no longer the remote source and well-spring of invention and creativity but, are part of the problem solving , problem identification and strategic brokering that characterize the knowledge industries

(Gibbons et al. 1994: 86)

Nowotny: building links and connecting

​

• Diffused expertise is a problem– people can believe that the knowledge that seeps across boundaries is expert knowledge – for Nowotny the solution is new ‘expert systems’

​

• ‘[E]xperts must now extend their knowledge, not simply to be an extension of what they know in their specialist field, but to consist of building links and trying to integrate what they know with what others want to, or should know and do.’

(Nowotny 2003: 155)

​

Can Researchers Avoid Intervening?

In conditions of modernity the social world can never form a stable environment in terms of the input of new knowledge about its character and functioning. New knowledge (concepts, theories, findings) does not simply render the social world more transparent, but alters its nature, spinning off in novel directions.

(Giddens 1990: 153)

​

Clarifying practices and incorporated into them

Social science research has the potential to illuminate and clarify the practices we are studying as well as the possibility to be incorporated into the very practices being investigated.

​

(Chaiklin 1993: 394)

​

The importance of ideas

Derry’s argument: Concepts are tools we use to examine the world. As we use them we make connections with what they mean in different ‘systems of inference’.

The task of teaching is not ever greater abstraction, but ever greater conceptual clarity.

​

My extension of her argument: One task of research is identifying and refining conceptual tools

Moore and Benington: co-creating public value

​

‘….[t]hrough closer linking of users and producers in creative joint development of products and services tailor-made to meet unmet human need.’

​

(Benington 2010: 45)

But how?

Marianne Cerf

• Not just participation
• Dialogical design processes on creating tools for sustainable agriculture
• Diagnosis of current tool use; testing new prototype; users work on their practices, examine the flexibility of the tool; researchers monitor the evolution of the tools, theorise etc.
• Recognising the different interpretations of the problem and the prototype

The Knowledge Practices Approach to User Engagement in Research

• Think of different practices as different epistemic cultures (after Knorr Cetina)

​

• With different purposes, histories and warrants for relevance, validity etc.

​

• Research projects become sites of intersecting practices

​

• Where what matters for each participant needs to be negotiated

​

Recognising Different Epistemic Practices: building common knowledge

• User engagement is not simply sharing the knowledge produced, or involving users as data gathers
• We need to identify what matters for each other: from completing the study within the time-scale – to producing ideas that are helpful and can be implemented
• Elsewhere I have called this common knowledge: consisting of ‘what matters’ or the motives of each participant
• Common knowledge then mediates collaboration

Making a Difference: working with users to develop educational research 2005-06

​

​

​

​

​

​

​

​

​

​

​

​

​

An ESRC seminar series with Judy Sebba – in the Teaching and Learning Research Programme

Having educational goals

​

Examining and following the anticipations of those who are working in the field of study

​

Building relationships with research participants from the beginning of research design

​

Building capacity among research users to engage in and with the research process

​

Keeping control of the study so that it also contributes to the research programme that produced it

From Reflexivity to Participation in Research Design

• Creating feedback loops (Edwards et al. 2010)

​

• University-led participatory research (design experiments)

​

• Combining small-scale studies (action research)

​

• Co-research for conceptual development (practice research; developmental work research; theories of change)

​

• User-led research (mental health studies in Birmingham; child-led research; foster carers as researchers)

​

A Case Study School

Implications of Participatory Models for Research Studies

• Build it into the design – it takes time

​

• Negotiate what is feasible to do with the participants at the outset

​

• Consider the demands of project management – holding it all together

​

• Funders need to recognise the additional costs

​

Implications of Knowledge Exchange for Departments and Faculties

• Recognising that knowledge exchange is not dissemination
• Capacity building in knowledge exchange at all levels of a research career
• Checking whether institutional drivers support knowledge exchange
• Considering the time/cost implications
• Examining the impact potential of secondments and research-based consultancies

​