NEEV Program Findings

​

Main Findings

• On an average attending live classes has benefited all groups of student, however the gain also depends on the student’s baseline score and the number of classes attended.

​

• We start to observe significant gain in outcome for students attending more than 50% of classes.

​

• Students who performed better in baseline also scored better in endline assessment.

​

• At mean level we observe that for students who attend more than 50% of classes, the gain in endline assessment with respect to baseline score is 64% and 85% higher in cohort 1 and cohort 3, respectively as compared to students who did not attend any live classes.

​

• Females constitute the majority of our sample. And on an average they have attended more number of classes. However, it cannot be conclusively said if females are benefiting more from the live classes.

NEEV Live Class Program Summary

Program Summary

• The program consists of At home math remediation live classes
• Grade 3rd-8th foundational concepts coverage were covered in these classes
• Classes were conducted 3 times a week
• Classes were taught by AF teachers via one-hour long classes with concept explanation, quizzes/polls, doubt solving

Scale

​

​

Assessment Summary

• 3 assessment cycles for 3 units taught
• Each assessment tested the key learning outcomes taught in the unit
• Each cycle consisted of:
• One baseline conducted pre-treatment
• One endline conducted post-treatment
• For operational ease, all schools were divided into 3 assessment cohorts and each cohort was tested for one unit

Offline vs. Online Assessments

• Mode of assessment affects students performance in an assessment
• In our analysis, performance of students in baseline assessment was 49% higher in offline assessments compared to online assessments. In the endline, this difference was 43%.
• Hence, in our analysis, we shall only be taking into consideration Cohort 1 assessment results, as well as Cohort 3 offline assessment results.

Population Description

*Analysis has been done only for students who gave assessments offline, to ensure the sample set is more representative of the overall population attending live classes

​

Population Description: Cohort 1

Distribution by Gender

​

Distribution by Academic Level

​

Distribution by Attendance

​

In a typical classroom of Cohort 1, out of all students who who gave baseline and endline:

1. Female students > males students
2. Taking BL scores as indicator of academic level, 38% students were in top 2 quintiles
3. 69% students out of all enrolled attended live classes, 23% attended >50% of classes

​

Population Description: Cohort 3 (Offline)

Distribution by Gender

​

Distribution by Academic Level

​

Distribution by Attendance

​

In a typical classroom of Cohort 3 (offline), out of all students who gave baseline and endline:

• Female students > males students
• Taking BL scores as indicator of academic level, 38% students were in top 2 quintiles
• 51% students out of all enrolled attended live classes, 27% of these students attended >50% of classes

​

​

Attendance Patterns

*Analysis has been done only for students who gave assessments offline, to ensure the sample set is more representative of the overall population attending live classes

​

Attendance Patterns: By Gender

Average attendance of students compared by gender

Access to Smartphone based on Gender and Geography

In both cohort 1 and 3, females are attending more classes compared to their male counterparts

• Females have attended 0.1 and 2.4*** classes more than males in cohort 1 and 3, respectively

​

Probably cause:

• The cause for male students not attending live classes is most likely NOT lack of device access. As per a separate Device Access survey conducted by Avanti, the % of students who had access to smartphones is similar across both genders.
• As per school teachers, this is more of a low motivation issue. However, this needs to be studied further.

​

Attendance Patterns: By Academic Level

Average attendance of students compared by academic level

For all students who attempted baseline and endline::

• Average attendance in cohort 1, 3.4 out of 13 classes (~26%)
• Average attendance in cohort 3, 3.6 out of 14 classes (~26%)

​

Taking BL scores as indicator of academic level:

• No patterns are emerging to relate academic level of students to their attendance levels
• There is a weak correlation between attendance and baseline score (correlation coefficient 0.11 and -0.05 in cohort 1 and 3)

​

Research question for 2022-2023 for qualitative analysis:

• Do students in higher quintiles tend to exit the program as it progresses?

​

Learning Gains Analysis:

*Analysis has been done only for students who gave assessments offline, to ensure the sample set is more representative of the overall population attending live classes

​

Mean and Mean Shift

Cohort 1 (Total marks = 20)

​

Cohort 3 (Total marks = 15)

​

• Effect of live classes in terms of mean shift is evident in both cohorts after students achieve an attendance level of more than 50% classes

​

• At mean level we observe that for students who attend more than 50% of classes, the gain in endline assessment with respect to baseline score is 64% and 85% higher in cohort 1 and cohort 3, respectively as compared to students who did not attend any live classes.

​

• Research question for 2022-23: Notice baseline score shift by attendance in cohort 3. By end of program, are academically gifted students moving out of program or attending less classes?

Regression Analysis: Effect of Live Classes

Cohort 1

​

Cohort 3

​

• Controlling for all other factors, effect of live classes becomes statistically significant at attendance levels above 50%
• Treatment effect values show the effect of live classes on endline scores, keeping all other factors constant. For e.g. in cohort 1, students who attended >50% to <75% classes scored 1.545 marks higher than control students in the endline.

Regression Analysis: Effect of Gender

Female students are scoring more in assessments and also attending more classes. However, we cannot conclusively say if the treatment has a higher impact on them.

• Females have attended 0.1 and 2.4*** classes more than males in cohort 1 and 3, respectively
• Females have scored 0.517 and 0.127 more than males in the baseline of cohort 1 and 3, respectively
• Females have scored 0.515 and 0.126 more than males in the endline of cohort 1 and 3, respectively

​

However, it cannot be conclusively said if females are benefiting more from the treatment since:

• In cohort 1, the increase in assessment scores is more in females compared to males by 0.339
• In cohort 3, the increase in assessment scores is lesser in females compared to males by 0.204.

​

Learning Impact by Academic Level

On an average, students who performed better in baseline also scored better in endline assessment.

• In cohort 1, after controlling for other factors, for each additional mark obtained by students in baseline assessment, they gained on an average about 0.5 additional marks in the endline. Total marks in this assessment were 20.

​

• In cohort 3, after controlling for other factors, for each additional mark obtained by students in baseline assessment, they gained on an average about 0.5 additional marks in the endline. Total marks in this assessment were 15.

Quintile Movement

Cohort 1, N = 246

Cohort 3, N = 152

Quintile movement is as shown in the accompanying charts:

​

In this population in cohort 1:

Overall of 61 out of 155 students (39%) who were below Q4 on the baseline moved forward to Q4 and Q5.

​

In this population in cohort 3,

39 out of 99 students (39%) did the same.

Quintile Movement

Cohort 1, N = 82

Cohort 3, N = 68

Based on findings of our descriptive statistics and regression analysis, if we consider only those students who have attendance levels above 50% as treated, then the quintile movement is as shown.

​

In this population in cohort 1:

Overall of the 20 out of 40 (50%) students who were below Q4 on the baseline moved forward to Q4 and Q5.

​

In this population in cohort 3,

27 out of 49 (55%) students did the same.