1 of 41

© UNICEF/UN0399491/Bukhari

How to measure delivery effectiveness beyond coverage?

8 July 2021

Andreas Hasman

On behalf of the UNICEF Delivery Effectiveness project team

2 of 41

  • Identify parameters and measures of delivery effectiveness
  • Evaluate the effectiveness of country VAS programs
  • Explore patterns in delivery effectiveness parameters across different strategies and contexts
  • Identify tools for countries to assess multi-parameter effectiveness for program planning and delivery

​

Project objectives

3 of 41

Identify parameters

Review literature

Consult stakeholders

Interview key informants

Identify measures

Analyze VAS data

Compare delivery strategies

Project timeline

4 of 41

Identifying effectiveness parameters

Primary Health Care Performance Initiative (PHCPI) conceptual framework

5 of 41

11 Survey parameters

Delivery effectiveness survey, 2021

Community awareness

Community acceptance

6 of 41

11 Survey parameters

Delivery effectiveness survey, 2021

Community awareness

Community acceptance

Sustainability

7 of 41

Community awareness

Access

Community acceptance

Sustainability

11 Survey parameters

Delivery effectiveness survey, 2021

8 of 41

Community awareness

Access

Community acceptance

Sustainability

Availability

11 Survey parameters

Delivery effectiveness survey, 2021

9 of 41

Community awareness

Access

Community acceptance

Sustainability

Availability

Service quality

11 Survey parameters

Delivery effectiveness survey, 2021

10 of 41

Community awareness

Access

Community acceptance

Sustainability

Availability

Service quality

Clinical outcomes

11 Survey parameters

Delivery effectiveness survey, 2021

11 of 41

Community awareness

Access

Community acceptance

Sustainability

Availability

Service quality

Clinical outcomes

Responsiveness

11 Survey parameters

Delivery effectiveness survey, 2021

12 of 41

Community awareness

Access

Community acceptance

Sustainability

Availability

Service quality

Clinical outcomes

Responsiveness

Equity

11 Survey parameters

Delivery effectiveness survey, 2021

13 of 41

Community awareness

Access

Community acceptance

Sustainability

Availability

Service quality

Clinical outcomes

Responsiveness

Equity

Efficiency

11 Survey parameters

Delivery effectiveness survey, 2021

14 of 41

Community awareness

Access

Community acceptance

Sustainability

Availability

Service quality

Clinical outcomes

Responsiveness

Equity

Efficiency

Resilience

11 Survey parameters

Delivery effectiveness survey, 2021

15 of 41

Community awareness

Access

Community acceptance

Sustainability

Availability

Service quality

Clinical outcomes

Responsiveness

Equity

Efficiency

Resilience

11 Survey parameters

Delivery effectiveness survey, 2021

16 of 41

Effectiveness parameter definitions

17 of 41

70 respondents completed the vitamin A delivery effectiveness survey

18 of 41

Respondents most frequently worked at the global or national level

70 respondents

Global

(n=36)

National

(19)

Subnational

(8)

Regional

(7)

Source: Vitamin A delivery effectiveness survey results, 2021

19 of 41

Respondents most frequently worked in immunization or nutrition

70 respondents | Nearly half of respondents (32) reported working in multiple focus areas

Immunization (n=33)

Nutrition (28)

Maternal & newborn health (19)

Malaria (15)

Polio (15)

HIV/AIDS & tuberculosis (11)

Neglected tropical diseases (9)

Sexual & reproductive health (9)

Other (17)

10

30

20

Source: Vitamin A delivery effectiveness survey results, 2021

20 of 41

At least three-quarters of respondents identified equity, service quality, access and sustainability as important for decision-making (n=70)�

Source: Vitamin A delivery effectiveness survey results, 2021

Yes, important for decision-making

Equity

Service quality

Access

Sustainability

Community acceptance

Availability

Clinical outcomes

Efficiency

Responsiveness

Community awareness

Resilience

7

11

14

16

19

24

24

24

26

28

32

n =

No, not important for decision-making

63

59

56

54

51

46

46

46

44

42

38

21 of 41

At least three-quarters of respondents identified equity, service quality, access and sustainability as important for decision-making (n=70)�

Source: Vitamin A delivery effectiveness survey results, 2021

Equity

Service quality

Access

Sustainability

Community acceptance

Availability

Clinical outcomes

Efficiency

Responsiveness

Community awareness

Resilience

n =

n =

Yes, important for decision-making

No, not important for decision-making

7

11

14

16

19

24

24

24

26

28

32

63

59

56

54

51

46

46

46

44

42

38

22 of 41

Equity, access and availability were most frequently ranked in the top half of parameters identified as important for decision-making (n=70)�

Source: Vitamin A delivery effectiveness survey results, 2021

Equity

Service quality

Access

Sustainability

Community acceptance

Availability

Clinical outcomes

Efficiency

Responsiveness

Community awareness

Resilience

n =

41

27

42

21

22

36

21

15

8

16

10

n =

No, not important for decision-making

Ranked as high priority

63

59

56

54

51

46

46

46

44

42

38

7

11

14

16

19

24

24

24

26

28

32

23 of 41

Community acceptance

5

Sustainability

2

Access

3

Equity

1

Availability

4

5 parameters identified for inquiry

24 of 41

Community acceptance

5

Sustainability

2

Access

3

Equity

1

Availability

4

5 parameters identified for inquiry

25 of 41

Identify inequality in vitamin A supplementation delivery programs

Example 1: Equity

​

OBJECTIVE

26 of 41

Poorest

Richest

Inequality

gap

Are traditional measures of VAS coverage equity sufficient?

27 of 41

Potential problems

with using existing measures of inequity

Multiple deprivations

Adverse health

outcomes

be risk exposures

Underlying distribution of deprivations

28 of 41

Have not received MCV1

5

Mother with no formal education

2

Household in poorest wealth quintile

3

No improved water source in household

1

Have not received DTP1

4

Sum of the number of deprivations experienced by children 9–35 months old

VAS Deprivation Score

29 of 41

Percentage of children that are zero-dose for VAS with 0–5 deprivations

Absolute difference in zero-dose VAS between groups (0–5 deprivations)

Relative difference in zero-dose VAS between groups (0–5 deprivations)

Distribution of deprivation scores among children

​

​

30 of 41

Percentage of children that are zero-dose for VAS with 0–5 deprivations

Absolute difference in zero-dose VAS between groups (0–5 deprivations)

Relative difference in zero-dose VAS between groups (0–5 deprivations)

Distribution of deprivation scores among children

​

​

31 of 41

Percentage of children that are zero-dose for VAS with 0–5 deprivations

Absolute difference in zero-dose VAS between groups (0–5 deprivations)

Relative difference in zero-dose VAS between groups (0–5 deprivations)

Distribution of deprivation scores among children

​

​

32 of 41

Percentage of children that are zero-dose for VAS with 0–5 deprivations

Absolute difference in zero-dose VAS between groups (0–5 deprivations)

Relative difference in zero-dose VAS between groups (0–5 deprivations)

Distribution of deprivation scores among children

​

​

33 of 41

Equitable

Low absolute difference of zero-dose VAS for children with deprivation scores of

0 compared to 5

​

Low relative

difference of zero-dose VAS for children with deprivation scores of

0 compared to 5

​

Underlying distribution has few children with deprivation scores >3

1

Moderately equitable

2

Low absolute difference of zero-dose VAS for children with deprivation scores of

0 compared to 5

​

High relative

difference of zero-dose VAS for children with deprivation scores of

0 compared to 5

​

Underlying distribution has few children with deprivation scores >3

​

Moderately inequitable

3

High absolute difference of zero-dose VAS for children with deprivation scores of

0 compared to 5

​

Low relative

difference of zero-dose VAS for children with deprivation scores of

0 compared to 5

​

Underlying distribution has children with deprivation scores >3

​

Inequitable

4

High absolute difference of zero-dose VAS for children with deprivation scores of

0 compared to 5

​

High relative

difference of zero-dose VAS for children with deprivation scores of

0 compared to 5

​

Underlying distribution has children with deprivation scores >3

34 of 41

​

Identify sustainability in vitamin A supplementation delivery programs

Example 2: Sustainability

​

OBJECTIVE

35 of 41

Vitamin A supplementation coverage, 2010–2018

Administrative semester VAS coverage data for 64 priority countries

Sustained, high admin VAS coverage data (n=15)

Nearly sustained, high admin VAS coverage data (n=3)

Unsustained and/or low admin VAS coverage data (n=32)

Unsustained, low admin VAS coverage data (n=13)

Source: Vitamin A delivery effectiveness survey results, 2021

Notes: TKM had no reported administrative data

​

36 of 41

Of the 15 countries with sustained coverage all nine years,

  • 12 (80%) used a predominantly non-routine delivery strategy
  • 3 (20%) used a mixed delivery strategy
  • None used a predominantly routine delivery strategy

​

37 of 41

Of the 15 countries with sustained coverage all nine years,

  • 12 (80%) used a predominantly non-routine delivery strategy
  • 3 (20%) used a mixed delivery strategy
  • None used a predominantly routine delivery strategy

​

38 of 41

Of the 15 countries with sustained coverage all nine years,

  • 12 (80%) used a predominantly non-routine delivery strategy
  • 3 (20%) used a mixed delivery strategy
  • None used a predominantly routine delivery strategy

​

39 of 41

  • Evaluate national-level programs
  • Compare programs across countries
  • Recommend delivery strategies in specific contexts
  • Plan national-level programs challenge: dependence on population-based surveys; focus

​

Applied learning

40 of 41

IMAGES

​

Child receiving vitamin A drops: UNICEF/UN0399491/Bukhari

Group of children pose together: UNICEF/UN0326753

Children walking arm-in-arm: UNICEF/UN056968/Ose

​

​

​

41 of 41