BOND evidence principles checklist

	B	C	D	E	F	G	H	I
1
2
3
4	Principles and checklist for assessing the quality of evidence
5
6
7
8
9	Contents:
10
11	This worksheet: Guide to the principles and checklist
12	1. What are the principles and the checklist?
13	2. Why are we developing them?
14	3. How have the principles and the checklist been developed so far?
15	4. How is this edition of the principles and checklist structured?
16	5. How to use the checklist
17
18	Worksheet 2:	The checklist (use with the scoring scale definitions which are on the next worksheet)
19
20	Worksheet 3:	The scoring scale definitions
21
22	Worksheet 4:	Your results
23
24
25	1. What are the principles and the checklist?
26
27	The principles and associated checklist have been developed as a tool for assessing the quality of evidence collected and used by NGOs to measure, learn from and demonstrate their impact.
28	They can be used to review and quality assure existing evidence (e.g. an evaluation report, research study or case study), and as a reference point when thinking about generating new evidence (e.g. to set an evaluation terms of reference).
29	The tool has been developed (and is maintained) by NGOs for NGOs in order to set a common standard for assessing evidence. It aims to build on NGO values, and set standards which are relevant and practical to the scale of NGO operations and the type of interventions NGOs deliver. In time, the tool may be developed to include an element of external verification and a ‘kite mark’.
30	The five principles against which evidence is assessed are: voice and inclusion, appropriateness, triangulation, contribution and transparency. For each principle the checklist has four questions that can be used to test the quality of the piece of evidence. Each question is scored on a 1-4 scale. An overall score and colour (amber, yellow, light green or dark green) is then assigned to each principle to provide a holistic picture of the overall quality of the evidence being assessed.
31
32	2. Why have we developed them?
33
34	All organisations strive to base their decisions on evidence of what works. NGOs are no different; they want to ensure they are putting their resources into the interventions that can make the biggest difference to the lives of poor and marginalised people. The challenge is that NGOs do not always have a strong evidence base for what works and how. This is especially the case for areas such as empowerment or accountability, which are difficult to measure. As a result NGOs are not always sure if they are basing decisions on solid evidence of what works.
35	Recognising this challenge, NGOs are looking to move forward in a way that is relevant and practical for them. NGOs need a standard of quality which is appropriate for the methods they use to generate evidence. Statistical methods have established standards of quality, but similar standards do not exist yet for the types of qualitative methods, or indeed other, non-statistical quantitative data collection methods that are commonly used by many NGOs in generating evidence. The checklist is an effort by UK NGOs to define a common approach to quality, taking into consideration the level of resources at the disposal of most NGOs and the types of interventions that they implement.
36
37	3. How have the principles and the checklist been developed so far?
38
39	Initial discussions on the checklist were held with a small group of Bond members in 2011, followed by a consultation with a larger group of 30 NGOs, Comic Relief, DEC and DFID in 2012. An initial list of principles and criteria was drawn up and piloted by seven NGOs (CAFOD, Christian Aid, EveryChild, the International HIV/AIDS Alliance, Self Help Africa, Traidcraft and World Vision) between May and June 2012. After a second consultation workshop with 28 NGOs this pilot draft of the criteria was created, containing five principles and twenty criteria. Following discussion with the pilot group and DFID in July 2012 the decision was taken to develop scales for each of the questions. The scales were drafted by a consultant and agreed at a workshop with the pilot organisations in August.
40	We then opened up the scoring scales for wider piloting throughout the sector for a 6 months period ending 31 May 2013. The learning from this exercise was used to further refine and improve the tool.
41
42	4. How are the principles and checklist structured?
43
44	The following section provides a brief explanation of each of the five principles on which the checklist is structured and why it is important to quality evidence.
45	1) Voice and Inclusion: the perspectives of people living in poverty, including the most marginalised, are included in the evidence, and a clear picture is provided of who is affected and how:
46	Ensuring that the voices of poor people, particularly the most excluded and marginalised, are captured and form the basis of any analysis of what has changed, for who, and how is key to generating compelling evidence. People living in poverty are the best judges of what has changed in their lives and their wider communities and need to be a key source for understanding the contributions of an intervention. The more they are involved in informing the approach to the evidence gathering, analysing and giving meaning to the data and validating findings, the richer and more compelling the evidence of what has changed in their lives will be.
47	2) Appropriateness: the evidence is generated through methods that are justifiable given the nature of the purpose of the enquiry:
48	Matching the right methods for data collection and analysis with the purpose of the enquiry is key to generating quality evidence. Methods need to be based on a clear understanding of what the enquiry aims to do. The methods being used to collect data need to be relevant and generate reliable data; the methods for analysis need to be systematic and support the generation of convincing conclusions, and the sample size needs to be in proportion to the conclusions that are sought and the level of validity needed. Also, the team carrying out the assessment needs to be made up of people with the required skills and characteristics.
49	3) Triangulation: the evidence has been generated using a mix of methods, data sources, and perspectives:
50	Talking to different groups of people, using different data collection methodologies, and collecting different types of data from different sources helps manage the risk of possible bias. Getting different stakeholders with different perspectives, such as beneficiaries, partners, and peers, to validate findings and conclusions will provide a vital check on their quality.
51	4) Contribution: the evidence explores how change happens, the contribution of the intervention and factors outside the intervention in explaining change:
52	Evidence of change needs to explore the causal links. It has to examine the contribution of an intervention, explore whether the assumptions about what causes what are accurate, and examine the relative influence of other factors. There needs to be a point of comparison to anchor any judgement of change. Also, the enquiry needs to be actively looking for the unexpected changes and explaining these in the analysis.
53	5) Transparency: the evidence discloses the details of the data sources and methods used, the results achieved, and any limitations in the data or conclusions:
54	Transparency and openness underpin strong evidence. It is not possible to assess whether evidence is accurate or representative, identify any gaps or potential bias, or assess the logical validity of the analysis, unless the data and the process used to collect and analyse it are clearly presented, including any problems and limitations.
55
56	5. How to use the checklist
57
58	The checklist appears in the next worksheet. Each of the five principles has four questions and each question can be answered on scale of 1-4 by placing an x in column 1, 2, 3 or 4. To aid with the scoring and to ensure consistency, scales have been developed for each question. The scales are defined in the third worksheet of this pack, and provide a short description of what practice looks like at each of the four levels.
59
60	For example:
61
62		1	2	3	4
63	1a. Are the perspectives of beneficiaries included in the evidence?	No beneficiary perspectives presented	Beneficiary perspectives presented, but not integrated into analysis	Beneficiary perspectives presented and integrated into analysis	Beneficiary perspectives presented and integrated into analysis, and beneficiaries have validated the findings; the evidence is strongly grounded in the voices of the poor
64
65
66	Scores for each of the questions are then added up and an overall score for the principles out of 16 is provided. Depending on the score, the principle is then assigned a colour: -
67
68	An overall score for the principle of 4 – 6			Evidence is weak in this area
69	An overall score for the principle of 7 – 10			Evidence meets a minimum standard in this area
70	An overall score for the principle of 11 - 13			Evidence meets a good standard in this area
71	An overall score for the principle of 14 – 16			Evidence meets gold standard in this area
72
73
74	Weighting Your Results
75
76	By default, each of the 5 principles is weighted equally (20%). To change the weight attached to the principles, you can adjust the percentages in the 'Weighting' column of the 'Your Results' worksheet (see example below). The total weighting must add up to 100%. A narrative space is provided to explain the reasoning behind your weightings.
77
78	Principle	Total score	Weighting	Weighted Score	Quality Assessment
79	1) Voice and Inclusion		20%
80	2) Appropriateness		20%
81	3) Triangulation		20%
82	4) Contribution		20%
83	5) Transparency		20%
84
85	The table below offers a worked through example of how the scoring would work for the 'voice and inclusion' principle:
86
87	Principle	Criteria	1	2	3	4	NA	Comments / evidence
88		1a. Are the perspectives of beneficiaries included in the evidence?				X
89	1) Voice and Inclusion	1b. Are the perspectives of the most excluded and marginalised groups included in the evidence?				X
90		1c. Are the findings disaggregated according to sex, disability and other relevant social differences?	X
91		1d. Do beneficiaries play an active role in designing the assessment process?		X
92		Score for voice and inclusion						11/16
93
94	Before entering your scores in the checklist and before completing the feedback worksheet, we recommend you do a 'save as' copy of the whole workbook and include your organisation's name in the title that you give it.
95
96	Now please go to the next worksheet (worksheet 2) where you’ll find the checklist. You will need to use it with the ‘scale definitions’ which are on worksheet 3.
97
98
99
100