Is digital reading generally more difficult? Nuances of effective reading behaviors for middle school readers��AERA 2023

Amanda Goodwin, Michael Havazelet, Sun-Joo Cho, Jorge Salas, and Matthew Naveiras

The research reported here was supported by the Institute of Education Sciences, U. S. Department of Education, through Grant R305A150199 and R305A210347 to Vanderbilt University. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.

​

​

What do you think?

​

Is digital reading harder than paper reading? Different? Similar?

​

When you read academically, what mode do you choose? Why? What tools or strategies do you use? Does your reading differ by mode? How?

2

Let’s think about our assumptions

• Generally: Prescriptive approaches to reading… do this, 1-path to reading success:
• We consider the tools and behaviors that middle school students use to support digital and paper reading comprehension.
• Generally: considering the dichotomy— is Digital Reading is Harder (or Worse?)
• Multiple syntheses and meta-analyses suggest readers tend to perform better on comprehension of paper-based texts compared to digital texts
• Reframe this discussion from a which is better (outcome focus) to how do readers orchestrate reading behaviors to comprehend (process focus) in different mediums..

3

Background

• Multiple syntheses and meta-analyses suggest readers tend to perform better on comprehension of paper-based texts compared to digital texts yet research is inconclusive (Coiro, 2021).
• Often, product (i.e., reading comprehension) is prioritized over process (i.e.., understanding what readers are doing during reading).
• Much variability to consider when studying digital reading (medium, devices, tasks, etc.)
• Most research considers single behaviors but little research explores how those behaviors are orchestrated together

4

Theoretical framing

• Theory of New Literacies (Leu et al., 2013)
• emphasizes how “both old and new elements of literacy are layered in complex ways” (p. 1157)
• suggests a Dual-Level theory of New Literacies
• lowercase (new literacies)
• uppercase (New Literacies).

5

Research Questions

1. Our quantitative study explores how a large sample uses highlighting to support reading.
1. We explore (a) patterns in highlighting and (b) relations between highlighting and reading comprehension for middle school readers reading a bound text digitally and on paper.
2. Our qualitative study explores the broader strategies used by a smaller sample of students (i.e., from different ability groups based on district-tests) when reading a NAEP passage digitally.

6

Participants

• Quantitative: N=370 (85 fifth graders, 78 sixth graders, 82 seventh graders, and 125 eighth graders)
• Qualitative: N=13 fifth graders, some at each level (struggling, medium, high)

7

Protocol

• Pretest, Reading NAEP passage, Post-test
• ~half reading digital, half paper (randomly assigned)
• Digital and paper tools provided & modeled
• highlighter, dictionary, annotation, etc.
• Purpose: Read like read in school
• iMotions

8

9

Quantitative Measures

• Outcome: Reading Comprehension
• 14 item posttest, marginal IRT reliability=0.81
• Highlighting
• length (#letters; #words)
• linguistic characteristics (word, sentence, sentence+phrase, paragraph, phrase, multiple sentences, year, proper noun, verb)
• content relevance (AOI)
• 16 AOIs, 1-19 words, average 7.81 words per AOI
• Covariates: pretest/content knowledge; preference; reading ability; demographics; look back; dictionary use

10

Quantitative Analyses

1. Explored patterns descriptively
2. Explored differences by medium
1. nonparametric test: ART ANOVA (aligned rank transform analysis of variance)
2. visual heatmaps
3. Explored links to comprehension
• explanatory IRT models with smooth functions to model linear and nonlinear effects of highlighting variables on comprehension

11

Qualitative Measures & Analyses

• Open Coding of 40 students to identify behaviors and create a codebook
• Purposive selection of fifth graders (N=13) and coded for:
• re-reading
• scrolling
• tool use
• embodied tracking
• off-task behaviors
• Visualization with of Action Paths and TimeScapes

12

Results

Big takeaways:

1. Results involving nonparametric tests indicate similarities and differences across mediums, and these patterns differ by part of text, yet few interactions between mediums and reader characteristics occur.
2. These similarities and differences were also present for comprehension. Here, most highlighting variables are nonsignificant predictors except highlights in areas of interest which support comprehension for both mediums controlling for other predictors and are particularly important for locate and recall items and for students with more content knowledge.
3. Qualitative results suggest multiple pathways to success (including non-expected tool use) and occasional re-reading as a potentially important behavior.

13

Quantitative Results- More paper highlighting

• out of 370 participants, 158 (42.7%) did not highlight on paper and 227 (61.3%) did not highlight digitally. When considering individuals, 85 students (23%) highlighted on paper but not digitally whereas only 18 students (4.9%) highlighted digitally but not on paper.
• When considering words, on average each word was highlighted 20.33 times on paper versus 11.21 times digitally, which according to a a non-parametric sign test was significantly different (p<.001).

​

14

Quantitative Results & Medium Differences

• Most at the phrasal level followed by highlighting at sentence level
• When paper highlighting, highlights also occurred at the word level, but not on average digitally.
• On average, highlights often included proper nouns and verbs with differences by medium. Highlights also included years.
• Much rarer were sentence+phrase, multiple sentences, and paragraph highlights.

15

Links to Comprehension

Multiple highlighting variables were significantly related to reading comprehension controlling for the other predictors in the model, but there were some differences by the part of text read.

​

For Paper: the more paper highlights at the sentence level or including proper nouns, the worse the performance on reading comprehension.

For Paper and Digital, the more highlights are in AOIs, the better the performance on comprehension with this being the case for both parts digitally and just for Part 2 on paper.

16

Qualitative Results- Trends between Behaviors and Comprehension

Action paths chart trends across ability groups and draw links between behaviors during the cold read and Q&A comprehension performance

​

​

​

17

Qualitative Results- Trends between Behaviors and Comprehension

Most behaviors were not linked with comprehension when considered in isolation

​

​

​

18

Qualitative Results- Trends between Behaviors and Comprehension

Exception:

Occasional rereading is associated with accurate look backs and increased performance in the Q&A

19

Qualitative Results- Readers’ Orchestration of Behaviors

Timescapes illustrate how individual readers orchestrate multiple behaviors

20

Qualitative Results- Readers’ Orchestration of Behaviors

Here, a student subverts expected use of the highlighter to track

​

What may seem as non-strategic tool use seems to serve their purpose and support comprehension

21

Discussion

• Great variability (even differences across parts of the text)
• Generally, re-reading and strategic highlighting (i.e., highlighting in AOIs) seems supportive
• No single path towards comprehension
• Many participants treating digital reading as paper; few exploring digital tools in novel ways

22

Now– what do you think?

​

When you read academically, what mode do you choose? Why? What tools or strategies do you use? Does your reading differ by mode? Is one harder or easier?

23

EXTRA SLIDES

For now, I’m moving excess slides after this one, not deleting

24

Qualitative Results– Multiple behaviors

Re-reading: linear reading, non-linear reading, and occasional re-reading

• 3 nonlinear
• frequently scrolled, skipping paragraphs, not ordered
• 5 read straight through
• 5 re-read
• 4 read phrases or sentences 2-5 times
• 1 re-read quickly at least 16 times

​

25

Qualitative Results– Multiple behaviors

Tool Use:

• no use of annotation or zom
• 4 used highlighter
• TS, names, date
• 1 highlighted over half the words in the text
• challenges with the highlighting tool
• 2 used define function
• low frequency words (propaganda, rescinded)
• non strategic (the, click on words at random)

26

Qualitative Results– Connections to Comprehension- Multiple Behaviors;

27

6 readers, contrasting how two readers from the same reading ability group orchestrated behaviors

• Low-Ability Readers: Nonlinear with No Tool Use v. Re-Reader With Minimal Tool Use
• Mid-Ability Readers: Linear with No Tool Use v. Re-Reader with Maximal Tool Use
• High-Ability Readers: Linear with Prolonged Mouse Tracking v. Re-Reader with No Tool Use

Qualitative Results– Multiple behaviors

Scrolling:

• 10/13 scrolled only functionally (1x or 2x at bottom)
• 3 made seemingly non-functional smaller scrolls

Embodied Tracking

• 4 students; mouse pointer to track words

Off-Task Behaviors

• 10 veered off task
• 7 brief breaks
• 3 extended breaks

28