1 of 28

Do Students Read Instructor Emails?

A Case Study of Intervention Email Open Rates

Link to Slides

Link to Paper

2 of 28

Authors

Elexandra Tran

Angela

Zavaleta Bernuy

Runlong �‘Harry’ Ye

University of Toronto

Naaz Sibia

Michael Liut

Bogdan Simion

Andrew Petersen

Abhijoy Mandal

Joseph Jay Williams

University of Toronto Mississauga

University of Stavanger

University of Toronto

University of Toronto Mississauga

Hammad Shaikh

2

3 of 28

Motivation & Background

01

3

4 of 28

Emotional Engagement

Programming self-efficacy and learning persistence
Enhanced with instructor interaction

Regular feedback
Monitoring group collaboration
Regular announcements

4

5 of 28

Course Communication

Instructors expect students to read emails.

But how do they know?

Instructors expect students to engage with emails.

But how do they know?

Communication tools
Email for instructor communication

Preferred
Common engagement strategy

Email interventions

Reminders for learning

5

6 of 28

Email for Instructor Communication

Instructors expect students to read emails.

But how do they know?

Instructors expect students to engage with emails.

But how do they know?

for a sequence of 3 emails in CS1

Email open rates and link click rates

Compare the rates to:

Instructors’ expectations
Students’ self-reported data

Planning emails: emails with planning prompts for starting homework early

6

We investigate instructor email open rates and link click rates for a sequence of 3 emails in CS1

The intervention was specifically selected as an example because sending students emails about upcoming deadlines can help set student expectations, encourage them to plan their work, and it can be a sign that instructors care about students’ learning and want them to perform well

Starting homework early can also help them have more time to interact with the material [6] which can help with emotional engagement in class

Emails appear to be the preferred way of communication between instructors and students and have been linked to greater emotional engagement. This motivates investigating how often students in computing interact with emails, considering that factors such as instructor support and course engagement can impact retention which is a concern in computing.

7 of 28

RQ1

RQ2

How often do computing students open the planning emails?

How do open rate for a sequence of email change across weeks?

7

8 of 28

Methodology

02

8

9 of 28

1,200

746

45%

36%

<5%

19%

Initial student enrollment

Student included in analysis (consented)

Students self-identified as men, women, no-binary.

Students who did not disclosed

9

10 of 28

Data Collection Timeline

Student survey: email usage and intention of course enrollment

Instructors’ perspectives: predictions on student email engagement

Email open rates: whether a student opened the email or not

Link click rates: whether a student clicked on at least one email link or not

⅔ treatment: email | ⅓ control: no email

Pre-lecture work: Monday

Post-lecture work:

Friday

pre-w

post-w

Tue

Wed

Tue

Wed

Thu

Fri

10

Start of Term

Email #1

Email #2

Email #3

Week 1

Week 5

Week 7

Week 10

End of Term

Week 12

11 of 28

Email Design

03

11

12 of 28

Email with Planning Prompts

12

Calendar Link

Add to their calendar an event to work on their homework

Assignment Link

Easy access to the homework handout

Implementation Intention

Reduce the gap from planning to start the homework and making it possible!

13 of 28

13

When will you next work on <Homework Module>?

Can you start earlier than last week?

Hi <Name>,

We care about helping you keep on track with your work, and minimizing your stress. Past students have found emails like this helpful.

Take a few minutes to plan: When will you next work on <Homework Module>? Is there anything that might help you start earlier?

Some students set a reminder or add an event to their calendar to help keep them on track. To make that easy, we're giving you a default event that you can change.

Click this link for an event that works with ical and outlook.

Click this link for Google Calendar.

Students have said starting earlier helps reduce stress, as they have extra time to think, or are able to use office hours or ask questions on <Discussion Board> when they get stuck.

Also note that Assignment X is now available and due on <Due Date>. If you haven’t yet looked at the assignment, when can you take some time to work on it? Click this link to access Assignment X.

From your <Course Code> instructors and the <Research Group>

P.S: Great job if you've already finished <Homework Module> for this week! We're just sending this email to students to give you extra encouragement to adopt good study habits, like starting early. If you have any questions, email <Research Group> at < Email>.

14 of 28

Results

04

14

15 of 28

Email Open & Link Click Rates

15

16 of 28

Email Open Rates: Previous Condition

16

17 of 28

Email Open Rates: Engagement

17

18 of 28

Instructors’ Perspectives

Frequently checks emails
Good organizational skills
Keen interest in learning
Active in the course
Taking the course as a requirement
Already starts the homework early

6/12 instructors expected more than 75% of students to open an email

4/12 expected over 50%

3/12 instructors expected more than 90% of students to click on links

4/12 expected around 50%
4/12 expected 30%

10/12 instructors think there is a particular type of students that will interact with emails more

18

19 of 28

Email Open Rates: Usage

No differences per email usage

19

20 of 28

Email Open Rates: Course Requirement

Students taking the course as an elective tend to open more emails.

20

21 of 28

Email Open Rates: Usefulness

Students that find the email not useful tend to open more emails.

I find receiving a reminder message useful.

21

22 of 28

Email Open Rates: Homework Start Time

No difference in open rate between those starting the homework early vs late.

22

23 of 28

Discussion & Takeaways

05

23

24 of 28

RQ1

RQ2

How often do computing students open the planning emails?

How do open rate for a sequence of email change across weeks?

Open rates are around ~60%, and click rates are very low (~5%).
Open rates decrease by week by around ~2%, while click rates drop lower.

The number of emails received does not affect future email open rate.
The initial decision of opening (or not) an email, predicts how they interact with the emails in the future.

24

25 of 28

Instructor Expectations

Frequently checks emails

Taking the course as a requirement

Starting the �homework early

Finding the email useful

No difference

OPPOSITE

No difference

OPPOSITE

25

26 of 28

Limitations

Students get A LOT of emails.
Different ways of interacting.

Engagement
Device

No email engagement does not mean no course engagement.
Instructor population.
Email privacy features.

26

27 of 28

Conclusion

Emails to conduct interventions in large CS courses
Email engagement: open rates and link click rates
Instructor predictions

May or may not align!

Factors that may impact email engagement
Students’ initial decision to open can predict future engagement

27

This paper aims to evaluate the use of emails to conduct interventions in large computing courses. We investigate how computing students engage with emails by analyzing data about email open rates, the proportion of links clicked, and student behaviours. These empirical results are contrasted with instructor predictions of student behaviour.

Combining instructor predictions with empirical results, we found that instructors may not always have accurate expectations of their students regarding email behaviours. We provide insight into which factors may impact email engagement, such as whether the course is taken as an elective or how frequently students check their university emails. Furthermore, we discovered that students' initial decision to open the email can predict their future levels of email engagement. In addition to providing insight into how to make decisions by using emails in this context, we believe it can provide a broader insight into how students would engage with email interventions in other settings.

28 of 28

Thanks!

Do you have any questions?

Link to Slides

Link to Paper

ACKNOWLEDGEMENT: The project is supported by NSERC (#RGPIN-2019-0696), ONR (#N00014-21-1-257), and the financial support from the University of Toronto Mississauga.

CREDITS: This presentation template was created by Slidesgo, and includes icons by Flaticon, and infographics & images by Freepik. Some figures are generated by OpenAI’s Dall-E 3 for illustrative purposes.

Do you want to collaborate?

Email me at angelazb@cs.toronto.edu

Do you want to see what happens next?�See you at SIGCSE ’24!