Evaluating Education for Social Learning and Sustainable Development in NRM Contexts
ELRC Faculty of Education | Rhodes University | Eastern Cape South Africa | Prof Eureta Rosenberg E.Rosenberg@ru.ac.za
Contents
Contents
What is evaluation?��Evaluation is a systematic process of determining of the merit, worth, value and significance of something, using agreed-upon criteria. (Adapted from Scriven,1991 and American Evaluation Association, 2014)
Contents
Roles of Evaluation
Accountability purposes (accounting for resources received)
Improving efficiency, effectiveness, outcomes and impacts
Learning and development (at project or programme level)
Learning and development (organisational level)
Learning and development (at national or system level, theory)
Decision-making (e.g. should intervention continue or not)
Communication and Promotion, Advocacy
Evaluation is a systematic process of determining of the merit, worth, value and significance of something, using agreed-upon criteria
The special case of Evaluation in Sustainability Education
Beck, U. 1992. Risk Society. Towards a new modernity. Sage.
Global SDGs (Sustainable Development Goals) as an Evaluative Agenda
Gender
Poverty
Hunger
Health & Wellbeing
Education
Water
Energy
Decent Work
Economy
Cities
Production &
Consumption
Inequality
Climate
Life in
Water
Life on
Land
Peace &
Justice
Global
Partnerships
SDGs as an evaluative tool in local depth inquiry
ESD and social learning processes evaluate the gap between where we want to be & where we are – with environment & sustainability issues that are complex, connected
Nexus matters of
concern
Creative commons work of ELRC Rhodes University , South Africa
Evaluation in/as/of ESD
O’Donoghue, R. Rosenberg, E. Joon, D. & Krah, J. (in press) Evaluation: A ‘nested game changer’ in ESD as evaluative processes of learning-led change.
Evaluation in/as/of ESD
Contents
Contents
Similar? Different?
A Range of Evaluation Approaches
Measurement (empiricism)
Constructivism, interpretivism; narrative approaches
Developmental and process-based evaluations
(5th) Theory based approaches
Participation and action research; Strategic Adaptive Management
Quanti-tative
Quali-tative
Mixed
Range of Evaluation Approaches: Each of these have associated methods.
Measurement (empiricism): Look for pre-test/post-test or other (1) experimental design; (2) surveys especially large scale, to compare variables to detect shifts
Constructivism, interpretivism; narrative: (3) Most Significant Change Stories (Davis & Dart); (4) Case studies
Process-based evaluations: (5) Appreciative enquiry; (6) developmental evaluation (MQ Patton), (7) Value added methodology (Wenger & Trayner)
Theory-based approaches: (11) Designing a theory of change; (12) realist evaluation; (13) realist meta-evaluation/ synthesis
Participatory and Interventionist Approaches: (8) Participatory evaluation, (9) action research (Lewin); (10) Reflective practice (Argyris & Schon)
Evaluating Education & Training Common models
Kirkpatrick, D.L. 1994. Evaluating Training Programs: The four levels. San Francisco, Berret-Koehler.
Impact
Behaviour
Learning
Reaction
“VCF” for evaluating multiple levels of learning in longer term, non-linear process with open ended outcomes�Wenger, Trayner and De Laat (2011); Wenger-Trayner and Wenger-Trayner, 2020
Evaluating Education & Training Value Creation Framework model
Value Creation Framework for evaluating longer-term, non-linear and open-ended processes of social learning
Wenger, Trayner and De Laat (2011); Wenger-Trayner and Wenger-Trayner, 2020)
Possible Data Sources
Registers
Questionnaires
New Forum
Meeting report
New Budget
Call records
New Staff
Our example: Evaluating with students in an ESD course
Example from EI4WS �(WRC, SANBI, Michelle Hiestermann)
# of people engaged in restoring land
# improvements in livelihood projects
Participatory monitoring:
Where is the programme working, for whom, and why? � How can we build on this?� � Where is it not working, and why? What should we try next?
Case study questions:
Across all catchments: �What are we learning about the mechanisms of livelihoods and land restoration? About transdisciplinary research?
What is the impact of social learning?
What are the principles for polycentric governance?
Meta-evaluation questions:
Tsitsa Project Combining Monitoring & Evaluation; Quantitative & Qualitative
Activity-based evaluation: Skills for enterprise development��
Layers of social reality and research
Actions
Meanings made, decisions, mechanisms
Social structures & cultural norms
What if reality�has layers?
Empirically experienced –the tip of the iceberg
Hidden but Actual
Potential (to also turn into ice) – also Real
Critical & social realists (Bhaskar, Archer, Sayer) argued that reality is stratified
A depth ontology allows evaluators to investigate more than what can be empirically observed, to also look at what reality must be like, for the empirical to appear
This includes looking for absences - what should be there but is not, and why? (critical realism)
Types of research (from Sayer, A., 2010, Problems of Explanation and the Aims of Social Science)