Running Head: Defining Online Reading Comprehension
Defining Online Reading Comprehension:
Using Think Aloud Verbal Protocols To Refine A Preliminary Model of Internet Reading Comprehension Processes
Donald Leu, University of Connecticut, David Reinking, Clemson University,
Amy Carter, Clemson University , Jill Castek, University of Connecticut, Julie Coiro, University of Connecticut, Laurie A. Henry, University of Connecticut, Jackie Malloy, Clemson University, Kathy Robbins, Clemson University,
Angela Rogers, Clemson University, Lisa Zawilinski, University of Connecticut
Abstract
This paper provides an overview of our work on the TICA (Teaching Internet Comprehension to Adolescents) project, funded by the Institute of Education Sciences, U.S. Department of Education. We present the methods and analytic approaches used to explore a central question: Which skills and strategies appear to be important for successful online reading comprehension? To address this question, we used think-alouds and verbal protocol analysis with 53 of the most skilled adolescent online readers selected from a population of 1,025 seventh-grade students within 12 diverse and economically challenged school districts in South Carolina and Connecticut. In South Carolina, we work in mostly rural school districts; in Connecticut we work in mostly urban school districts. These populations were selected because they are most at risk to be left behind in an age of online information should they drop out of school (Finn, 1989; Friedman, 2005; 1993;Wylie & Hunter, 1994) and because this age group is especially engaged in using the Internet (Chandler-Olcott & Mahar, 2003; Lankshear & Knobel, 2003; Reinking, 2001).
The 53 seventh-grade students who provided think aloud data were selected on the basis of Internet use survey scores, which characterized them as the highest scoring students in online reading comprehension. They also met several other criteria: 1) frequency of Internet use, in and out of school; 2) diversity of Internet use, in and out of school; 3) expertise in explaining the strategies employed during online reading comprehension; and 4) comfort in explaining strategies to an adult. Following selection, these 53 students also participated in a short online reading task to verify that they were skilled users of the Internet.
We used think-aloud methods recommended by Afflerbach (2002) and Pressley & Afflerbach (1995) to elicit strategy use during online reading comprehension. During each of three sessions, participants were asked to read online and to think aloud, using both researcher-selected and student-selected reading assignments. Students' online reading sessions were recorded using Camtasia software (http://www.techsmith.com/camtasia.asp). This tool created a real-time movie of all online actions on the screen as well as an audio recording of verbal think-aloud data.
Data from Camtasia recordings of online screen reading and transcripts of students' think aloud comments were coded and analyzed using both constant-comparative (Bogdan & Biklen, 2003; Merriam, 1988) and abductive (Onwuegbuzie & Leech, 2006) methods. NVivo analytic tools were used to search for patterns across students’ online reading actions and verbal reports of online strategy use. Frequently used skills and common patterns of strategy use during online reading comprehension were then categorized into larger skill sets (e.g., defining a question or problem, locating information, critically evaluating information, synthesizing information, communicating information) and used to inform a taxonomy of online reading comprehension skills demonstrated by proficient online readers.
This paper presents the procedures used for conducting these think-aloud verbal protocols, using illustrative examples and online links to video examples (http://www.newliteracies.uconn.edu/iesproject/videos/). It also reports a preliminary description of the skill set revealed among adolescents assessed to be the most proficient in reading for information on the Internet. Finally, the authors share the website created for this project, making all of these procedures and methodological tools available to other researchers (see http://www.newliteracies.uconn.edu/iesproject/documents.html).
This paper is available online at: http://docs.google.com/Doc?id=dcbjhrtq_10djqrhz
Please do not quote without permission from the authors.
Presented at the Annual Meeting of the American Educational Research Conference. Chicago, IL. April 9, 2007.
The research
reported here was supported by the Institute of Education Sciences,
U.S. Department of Education, through Grant R305G050154 to the
University of Connecticut and Clemson University. The opinions
expressed are those of the authors and do not represent views of the
Institute or the U.S. Department of Education.
______________
Defining Online Reading Comprehension:
Using Think Aloud Verbal Protocols To Refine A Preliminary Model of Internet Reading Comprehension Processes
This paper provides an overview of our work on the TICA (Teaching Internet Comprehension to Adolescents) project, funded by the Institute of Education Sciences, U.S. Department of Education. It also describes the procedures we use to determine which skills and strategies appear to be important for successful online reading comprehension. Finally, it provides a preliminary description of the online reading comprehension skills and strategies that are emerging in our work. This work is aimed at refining definitions of reading comprehension and at informing literacy instruction in this area. Our work is also aimed at students who have relatively little access to the Internet (i.e., students in diverse and economically challenged school districts) and who are at risk of not acquiring the skills, strategies, and dispositions necessary for negotiating information and learning on the Internet. Our work is predicated on data clearly indicating that the Internet has become an essential context for reading comprehension in the twenty-first century.
The Internet Is This Generations' Defining Technology For Reading
The Internet has become a vital new dimension of reading in schools (International Reading Association, 2002; National Center for Education Statistics, 2003), in daily life (Lebo, 2003), and in the work place (Friedman, 2005; U. S. Department of Commerce, 2002). In 2005, for example, over one billion people, one sixth of the world’s population, were reading on the Internet (Internet World Stats: Usage and Population Statistics, n.d.). Most of this growth took place in the past five years (de Argaez, 2006). At this rate of growth, nearly half of the world's population will be reading online within five years.
This historic and global change in the technology of reading is especially pronounced among adolescents, redefining the social practices of literacy (Coiro, Knobel, Lankshear, & Leu, in press). Consider, the case of Accra, Ghana, for example, where a recent study (Borzekowski, Fobil, & Asante, 2006) indicates that 66% of 15-18 year olds attending school, and 54% of 15-18 year olds not attending school, report having gone online previously. In Japan, 98% of homes have access to the Internet with broadband 16 times faster than that found in the U.S. (Bleha, 2005). In the U.K., 74% of children and young people aged nine to nineteen have access to the Internet at home, and most of these (84% in all) are daily or weekly Internet users (Livingstone & Bober, 2005). In the U.S., 87 percent of all students between the ages of 12 and 17 report using the Internet, nearly 11 million do so daily (Pew Internet and American Life Project, 2005). Similar patterns are commonly found in other nations. These data suggest that the Internet is rapidly becoming a defining technology for literacy and learning, especially for adolescents (Alvermann, 2006; Reinking, 2001).
The Internet Requires Additional Online Reading Comprehension Skills And Strategies
Traditionally, we have tended to assume that online reading comprehension is isomorphic with offline reading comprehension (Coiro, 2003; Leu, Zawilinski, Castek, Bannerjee, Housand, Liu, & O'Neil, in press). Data are appearing, however, to question this assumption; online reading comprehension appears to require additional and perhaps somewhat different comprehension skills and strategies. One study, among sixth-grade students proficient at using the Internet (Coiro & Dobler, 2007) found that online reading comprehension shared a number of similarities with offline reading comprehension, but online reading comprehension was also more complex and included notable differences. A second study found no statistically significant correlation between scores on a state reading comprehension assessment and offline reading comprehension (Leu, Castek, Hartman, Coiro, Henry, Kulikowich, & Lyver, 2005). The results from this second study also suggested that new skills and strategies may be required during online reading. A third study (Coiro, 2006), using a regression model, found that although offline reading comprehension and prior knowledge contributed a statistically significant amount of variance to the prediction of online reading comprehension, additional, independent variance was contributed by knowing students’ online reading comprehension ability.
All of these studies point to the conclusion that additional skills and strategies are required during online reading comprehension. The influential report of the RAND Reading Study Group (2002) captures the essence of the problem: ". . . accessing the Internet makes large demands on individuals’ literacy skills; in some cases, this new technology requires readers to have novel literacy skills, and little is known about how to analyze or teach those skills” (p. 4).
Students Who Require Our Assistance The Most May Be Receiving It The Least
The mistaken assumption that online reading comprehension is identical to offline reading comprehension may produce a false sense of security that conventional instruction is preparing students for the literate world of today. This, in turn, may mitigate the urgency of finding instructional interventions that prepare students for that world. The assumption of isomorphic reading comprehension (Leu, et. al., in press) is most visible, perhaps, in U.S. state reading assessments. Not a single state reading assessment, required by No Child Left Behind (NCLB) legislation (No Child Left Behind Act of 2001 (2002), measures students’ ability to read and comprehend information online; no state assessment of reading measures students' ability to read and interpret search engine results, to read a web page to select the most likely link where information might appear, to critically evaluate information online for reliability or accuracy, or to read information contained in email messages, blogs, and wikis (Leu, Ataya, & Coiro, 2002). Thus, many of the important aspects of online reading comprehension are never assessed. As a result, unfortunately, they are seldom taught.
Indeed, No Child Left Behind (NCLB) legislation (No Child Left Behind Act of 2001, 2002) may actually be exacerbating the very problem it seeks to solve (Henry, 2006). This may be especially true for those students who require our support the most -- students in economically challenged school districts who have Internet access at home the least. Economically challenged school districts in the U.S. are now under the greatest pressure to raise reading test scores (Center on Education Policy, 2006), none of which measure online reading comprehension. These districts are forced to focus limited resources on raising performance on state assessments that solely evaluate offline reading comprehension; they have little time or interest in providing instruction in online reading comprehension skills (Henry, 2006). In more privileged districts, however, students have greater access at home and attend schools that provide more opportunity to use the Internet, since the pressure from NCLB to raise offline reading comprehension scores is substantially less (Center on Education Policy, 2006; Henry, 2006). The cruelest irony of NCLB may be that we end up helping those the least who require our assistance the most.
What Are The Skills And Strategies Required For Successful Online Reading Comprehension?
Currently, there is relatively little understanding of how reading comprehension instruction should be conceptualized or conducted in relation to online information (Castek, 2007; Coiro, 2003; International Reading Association; RAND Reading Study Group, 2002; Reinking, 1997). What are the skills and strategies required for successful online reading comprehension? This important question is the initial focus of our work.
To contextualize our study of this issue, we first present an overview of our entire project. Next, we describe the multiple theoretical perspectives used to frame our investigations, arguing that the issues and the contexts associated with online reading comprehension instruction are far too complex for any single perspective to account for all that is taking place. Then, we present the methodological approach we have taken to identify the major skills and strategies associated with successful online reading comprehension, providing examples of the strategies reported by our student informants. We also present an emerging taxonomy of online reading comprehension skills and strategies. Finally, we share several videos of think-aloud verbal protocols collected during online reading comprehension tasks to illustrate this approach and the strategies that students use during online reading comprehension.
An Overview Of The TICA Project
What are the skills and strategies required for successful online reading comprehension? How can and might they be taught? These questions are the focus of this three-year reading comprehension research grant, funded by the Institute of Education Sciences, U.S. Department of Education. Research teams from Clemson University and the University of Connecticut are conducting this work. The research takes place in seventh-grade classrooms within diverse and economically challenged districts in South Carolina and Connecticut. In South Carolina, we work in largely rural school districts; in Connecticut we work in largely urban school districts. These populations were selected because (a) they are most at risk to be left behind in an age of online information should they drop out of school (Finn, 1989; Friedman, 2005; Wylie & Hunter, 1994), (b) they appear to have fewer opportunities to develop these skills because of the nature of state assessments and pressure from NCLB (Henry, 2006), and (c) they represent an age group that seems especially attracted to the use of the Internet (Chandler-Olcott & Mahar, 2003; Lankshear & Knobel, 2003; Reinking, 2001).
The primary goal of Year 1 was to develop a theoretical, data-driven framework for producing high levels of online reading comprehension, engagement, and school learning among 7th grade students in economically challenged school districts. During Year 1, we collected and analyzed two sources of information to inform subsequent work: a) a survey of students in our target population aimed at characterizing Internet use at home and at school; and b) verbal protocol data obtained with think-aloud procedures from skilled, high volume Internet readers in our target population as they read informational texts obtained from the Internet.
In Year 2, the current year of the project, we are conducting a formative experiment (see Reinking & Bradley, 2004) of Internet Reciprocal Teaching (IRT), an instructional framework informed by Reciprocal Teaching (Palincsar, 1986; Palincsar & Brown, 1984). During this year, we seek to understand what factors enhance or inhibit the effectiveness of this instructional intervention, how it might be modified on the basis of those factors to increase its effectiveness in reaching our pedagogical goals, what unexpected, collateral effects it produces, and if and how it might transform the instructional environment for literacy instruction. Our primary goal is to determine how best to increase online reading comprehension among adolescents at risk of becoming dropouts. Data collected and analyzed during the formative experiment in Year 2 will inform Year 3 which will be a conventional controlled experiment. For example, Year 2 will inform us about the most promising way to implement the intervention across diverse contexts and what variables will be controlled or studied.
The major goal of Year 3 is to conduct an experiment with random assignment of treatment conditions at the teacher level using hierarchical linear modeling (HLM) procedures. The proposed experimental design will test the effects of our adapted reciprocal teaching approach designed to increase online reading comprehension and knowledge of effective Internet reading comprehension strategies. The experiment in the third year will span most of the academic year.
The Importance of Multiple Theoretical Perspectives
Theoretical orientations toward the changing nature of literacy on the Internet typically reflect a single tradition of inquiry or a single theoretical lens from sociolinguistics, psycholinguistics, information science, cognitive theory, or sociocultural theory (Coiro, Knobel, Lankshear, & Leu, in press). To develop viable approaches to online reading comprehension, we believe it is essential to bring multiple theoretical perspectives to bear, recognizing the multiple realities (Labbo & Reinking, 1999) that exist. Research questions related to online reading comprehension occur in contexts that are far too complex for any single perspective to account for all that is taking place.
Thus, we initially framed this project around the idea that the complexities of contending with digital literacies are best understood by adopting rich, complex, and multiple theoretical lenses (Labbo & Reinking, 1999; Coiro, Knobel, Lankshear, & Leu, in press). We conceptualize our work within a new literacies of the Internet perspective (Leu, Kinzer, Coiro, & Cammack, 2004), a perspective that integrates theoretical views from information science, cognitive science, sociocultural theory, critical theory, mulitliteracies, and sociolinguistics. In addition to a preliminary set of principles informing us about how the new literacies of the Internet are likely to be acquired, this perspective identifies five processes that take place during online reading comprehension:
“The new literacies of the Internet and other ICTs include the skills, strategies, and dispositions necessary to successfully use and adapt to the rapidly changing information and communication technologies and contexts that continuously emerge in our world and influence all areas of our personal and professional lives. These new literacies allow us to use the Internet and other ICTs to identify important questions, locate information, critically evaluate the usefulness of that information, synthesize information to answer those questions, and then communicate the answers to others.”
(Leu, et. al, 2004, p. 1572)
We also draw upon important insights generated from dual coding theory (Pavio, 1990), cognitive flexibility theory (Spiro & Jehng, 1990), multiliteracies (New London Group, 1996), and other theoretical frameworks as they help us to enrich our understanding of the data that emerges from our work.
Methods For Identifying Online Reading Comprehension Skills and Strategies
A central goal in our research is to develop a preliminary taxonomy of the skills and strategies that appear to be important for online reading comprehension. This taxonomy is important to our subsequent work, especially the instructional treatment study in Year 3. Although we could have selected participants from a population of highly-skilled, adult, online readers or students from higher achieving, and more economically privileged school districts, we determined that grounding this aspect of the work in our target population of students at-risk of dropping out of school had several advantages. Using adults or advantaged students who were highly proficient, but outside of our target population, may have provided us with skills and strategies that might not have matched the full range of needs among students in our schools. Thus, we sought informants who were the highest performing online readers in the schools we were studying, middle schools in diverse and economically challenged districts in South Carolina and Connecticut. We invited these students to serve as informants using think aloud procedures (Afflerbach, 2002; Pressley & Afflerbach, 1995).
Using An Online Survey To Select Our ParticipantsOur initial sample included a total population of 1,025 seventh graders in 14 middle schools (eight in South Carolina and six in Connecticut) whose parents completed permission forms and who then completed an online survey of Internet use (Henry, Mills, Rogers, & Witte, 2006). This survey of Internet use was designed to help us determine approximately 50 of the most skilled and experienced online readers in our total population to serve as our informants. The survey measured: a) online reading comprehension ability, b) frequency of Internet use, c) diversity of Internet use, d) expertise in explaining online reading comprehension strategies, and e) comfort in explaining strategies used when reading on the Internet to an adult. Each area of the survey is described in detail below, followed by the procedures used in selecting the student informants used in this portion of our work. The entire survey may be viewed at: http://camss.clemson.edu/READING/servlet/Page1.
Online Reading Comprehension Ability. We sought students who were skilled in online reading comprehension. Online reading comprehension ability was calculated from three different types of elements on a survey of Internet use, comprising a total of 16 items. Seven items consisted of rubric-scored performance items which evaluated the quality of the strategies students used during online reading tasks such as evaluating the reliability of information on the Internet, locating information for a given purpose, or downloading an email attachment. Each was scored from 0-2 with 0 representing "no response" or "incorrect response," 1 representing a "somewhat skilled response," and 2 representing a "highly skilled response." Five items consisted of multiple choice selections which evaluated skills such as reading a set of search engine results to determine the best choice for an information problem. Each multiple choice item was scored as 0 for "incorrect" and 2 for "correct." Four items consisted of responses about how frequently students reported evaluating information they read on the Internet at home or at school. Each was scored from 0-2 with 0 representing a response of "never" or "less than once a week," 1 representing a response of "once a week" or "a few times a week" and 2 representing a response of "once a day" or "several times a day." The highest possible total score for our measure of online reading comprehension ability was 32. The mean score for online reading comprehension ability among the total population was 12.14 (S.D. = 5.15). Quartiles were developed from the entire set of scores. Scores in the top quartile ranged from 17-27.
Inter-rater reliability for the rubric-scored responses was calculated using Fleiss' Kappa (1971), a statistic for measuring reliability when coding qualitative/categorical variables using multiple raters. A random sample of 50 sets of student responses from both research sites was derived for the seven skill items to be scored and the kappas for each scoring rubric was calculated among four independent raters. Fleiss' Kappa has a range of 0 – 1.00 and, since four raters were used, a kappa of .7 or better was desired for the purposes of this investigation. Reliability for the seven skill item rubrics on these 50 sets of survey responses ranged from .75 to .96, which was deemed acceptable. The four raters then divided the data set and scored the remainder of the student responses independently, using these same scoring rubrics.
Frequency of Internet use. We also sought students who used the Internet frequently. Twenty-seven survey questions asked students to report time spent on a variety of Internet activities in school and another, parallel, set of twenty-seven items asked students to report time spent on the same tasks outside of school. Students were asked to report on the frequency with which they performed Internet activities such as: "use search engines to locate information for a school assignment;" "read websites to help with short homework assignments;" "check a second source to see if the online information I found is true;" "look to see who wrote the online information I found before I use it;" and "play online games." Students rated their Internet use on a 6-point Likert scale: 0=Never, 1=Less than once a week; 2=Once a week; 3=A few times a week; 4=Once a day; and 5=Several times a day. Scores were totaled across in-school and out-of-school use to identify students who reported using the Internet most frequently. The total possible score on these items was 270, representing a student who reported doing each item on the survey several times a day, both in school and out of school. Total frequency of use scores ranged from 0 - 178. The mean score was 57.56 (SD = 34.93). Scores for frequency of use were divided into quartiles.
Diversity of Internet use. We also sought students who used the Internet for diverse purposes. The same survey questions used to determine the frequency of use on various Internet tasks were also used in a diversity of use analysis. If a student responded to any of the 27 survey items in the frequency of use analysis with "never," they received a score on that item of 0. If they responded with "Less than once a week" or a response indicating a more frequent use, they received a score of 1. A score of 1 indicated that they used the Internet for this particular purpose. The scores on these 54 items were then summed to derive a diversity of use score. The total possible Diversity of Internet Use Score was 54. Scores ranged from 0-50. The mean was 23.43 (SD = 9.16). Scores were then divided into three levels to indicate low (1-18), moderate (19-35), and high (36-54) general diversity of use. A rating of 0, 1, or 2 was assigned to each participant to indicate their level of general diversity where 0 = low, 1 = moderate, and 2 = high diversity of use.
Expertise in explaining online reading comprehension strategies. We also sought students who could provide a clear explanation about the strategies they use when reading on the Internet. One survey question was used to determine student expertise with providing a clear explanation about strategy use. This question evaluated students' ability to explain strategy use while reading and communicating on the Internet. The item asked: "You are looking for reliable websites about the rain forest. Your friend has sent you this list of four website addresses with no other information. If you had to predict which link would lead to the MOST reliable information about rain forests, which link would you choose? Why did you choose this answer?" The written explanation for this item was rubric scored with a possible score of 0 (no response or an incorrect response), 1 (a somewhat skilled response), and 2 (a highly skilled response). The score on this item was used to represent expertise in explaining online reading comprehension strategies. The total possible score was 2. Scores ranged from 0 - 2. The mean was .71 (SD = .70).
Comfort in working with an adult and thinking aloud. Finally, we sought students who would be comfortable with thinking aloud in front of an adult while reading online. The survey included this question, "Rate how comfortable you would be explaining to an adult (or thinking aloud) about where you go and how you read on the Internet." Responses included: Not at all comfortable (0), a little comfortable (1), somewhat comfortable (2), and very comfortable (3). The total possible score was 3. Scores ranged from 0 - 3. The mean was 2.04 (SD = .91).
Procedures for Student Selection To select our think aloud informants, the primary selection criterion was online reading comprehension ability. We only included students who scored within the top quartile of scores on this variable (17-27 out of 32). Then, within this population, we excluded students who: scored in the bottom quartile on the frequency of use score; had a low (0) Diversity of Internet Use Score; had a score of 0 on the item used to measure expertise in explaining online reading comprehension strategies; and had a score of 0 on the item used to measure comfort in working with an adult and thinking aloud. Parental permission was sought from all 63 students who met these criteria. Fifty-five out of sixty-three permission forms were returned. Two students decided to stop participating subsequent to the first think aloud session. This left a final sample of 53 informants (25 from Connecticut and 28 from South Carolina) for the three think aloud sessions. A comparison of the mean scores of the selection variables between the total sample and the informants appears in Table 1. The mean scores indicated that our think aloud informants, on average, obtained about two-thirds of the maximum score on the assessment of online reading comprehension, used the Internet for each of the 27 purposes in our survey about once a week, averaged a moderate to high level of using the Internet for diverse purposes, demonstrated greater than "somewhat skilled" responses in explaining online reading comprehension strategies, and were more than "somewhat comfortable" in working with an adult and thinking aloud about their reading. Our student informant population presented a range of performance levels with online reading comprehension which provided us with the opportunity to contrast how highly skilled readers performed with how readers performed who were capable, but not quite as skilled.
--------------------------
Table 1 Here
--------------------------
Sequence of Think Aloud Tasks
Our student informants completed three different online reading sessions, lasting from 40 minutes to an hour, in the spring of the school year. Each session consisted of a short interview and a series on online reading comprehension activities during which the student informants were asked to think aloud while reading on the Internet; the first session also contained a short training session on thinking aloud while reading online. The three sessions provided online reading comprehension tasks that varied along two dimensions.
One dimension involved the extent to which students completed researcher-determined versus student-determined online reading comprehension tasks. Session 1 provided students with a researcher-determined online reading task. Session 2 included both: a researcher-determined tasks as well as an activity where students were able to select tasks from a list. In Session 3 students engaged in an online reading comprehension activity that was entirely student-determined, since students brought their own questions to the Internet that they wanted to explore while reading.
The second dimension involved variation in how active the researcher was in prompting the think-aloud task, ranging from uninterrupted thinking aloud to a video-prompted think aloud. There is an important trade off in the methodological choices researchers must make with think-aloud, verbal protocols (Afflerbach, personal correspondence). To the extent that a researcher does not interrupt reading during think-aloud protocols, data are obtained that are less likely to be influenced by a researcher's prompts or probes, but fewer data are likely to be generated and it is not always at locations of interest to the experimenter. To the extent that a researcher prompts thinking aloud, more data are obtained and it can be obtained at important locations for building theory. However, additional data comes at the potential cost of being influenced by an experimenter's probes. We sought to obtain both unprobed and probed think aloud data and to make are decisions carefully and systematically.
In the first session, students completed an uninterrupted think aloud activity on a researcher-determined reading task. In the second session, students completed a structurally prompted think aloud activity, being prompted at fixed structural locations to think aloud by the experimenters while reading online. This took place within both researcher-determined and student-determined reading tasks. In the third session, students completed a think aloud activity followed by a video stimulated think aloud activity while viewing the video of their online reading. This took place within a student-determine reading task. By varying both the nature of the reading activity (from teacher-determined to student determined) and the nature of the think aloud activity (from uninterrupted think aloud to structurally prompted think aloud to video stimulated think aloud) we sought to collect strategy use data within a wide range of possible online reading and think aloud conditions.
Session 1: An Uninterrupted Think Aloud. The first session used an uninterrupted think-aloud procedure to provide us with baseline data on how students would perform if they were not interrupted by prompts, but rather spontaneously thought aloud while reading. Following a short training session on thinking aloud, students were given the targeted online reading comprehension task for session one along with a prompt to encourage them to think aloud throughout the session ("Can you tell me what you are thinking?"). Each student received a short, fictitious, message from another class, asking them to locate and evaluate the reliability of a “spoof site,” Save the Northwest Pacific Tree Octopus (http://zapatopi.net/treeoctopus/), provide three reasons for their answer, and summarize the most important information from that site in one or two sentences. They were asked to send their response via IM, email, or a post at a designated blog. Following the activity, students were interviewed to ensure that they were familiar with the term "reliable," an important concept for understanding the task. When asked what this term meant, all informants reported meanings to indicate that they understood the term (e.g., "It means that you can trust it;" "It means it will always be there for you;" or "It's like a friend that you can trust.") The entire session was recorded using Camtasia software, a tool that records a video file containing all screen movement as well as audio data from the interview and think aloud. The complete protocol used in Session 1 is available at: http://www.newliteracies.uconn.edu/iesproject/documents.html
Session 2: A Structurally Prompted Think Aloud. For the second session, each student was randomly assigned to one of two different online reading tasks: (a) evaluate two
websites for reliability, explain your evaluation in a word document
that is sent by email and then, if time, revise an entry on wikipedia
and explain how you figured out what to do in a word document sent as a
word attachment or (b) choose
two questions from a set of eight at a blog site
(http://newliteracies.typepad.com/subject_surfi/), use the Internet to
research them, and post the answers at the blog. The entire session was recorded using Camtasia software. The protocols used in Sessions 2a and 2b are available at: http://www.newliteracies.uconn.edu/iesproject/documents.html
Both tasks required students to think aloud ("Can you tell me what you are thinking?") at three types of structural locations, during the online reading comprehension task: (1) when students were reading any web page, (2) when students were about to click on a link and make an interactive decision about what to read next, and (3) when students were entering key words into a search engine. This resulted in students being prompted to explain their thinking at structural points such as the following:
• reading and choosing the questions to respond to;
• reading any web page;
• reading search engine results;
• reading about information to establish a site’s reliability;
• reading blog entries;
• reading to figure out how a blog works and where to post a response;
• reading while composing a blog response;
• selecting a search engine;
• reading, just before clicking on a search engine result;
• reading, when students were about to click on ANY link;
• reading, when students were entering search engine key words; and
• reading, when students selected the link to the compose section of the blog.
Session 3: A Video Stimulated Think Aloud Protocol. The third session provided students with an opportunity to read online about a topic that was of personal interest in a student-determined online reading comprehension activity. At the end of Session 2, each student was asked to come to the next session with an important question that they were interested in reading and learning more about on the Internet. They were also asked not to use the Internet to read about this question until the upcoming session. Twenty-five students were randomly assigned to an uninterrupted think aloud condition, following procedures used in Session 1; the other twenty-five students were randomly assigned to a structurally prompted think aloud condition, following procedures used in Session 2. After each informant determined that the online reading comprehension activity was completed, they also completed a video stimulated think aloud, using the video of the session they had just completed. The Camtasia video and audio recording of each student's reading session was played back and students were invited, using structurally prompted think aloud procedures, to explain what they were thinking at each important structural location. The complete protocol used in Session 3 is available at: http://www.newliteracies.uconn.edu/iesproject/documents.html.
Camtasia Video Recording of Online Screen Reading and Think Aloud Data
Throughout all sessions, Camtasia software was used to record all screen activity on the computer and all audio spoken by the student and researcher. Thus, video files for all sessions were recorded and saved on each computer used in the online reading and think aloud sessions. Each video and audio file was transcribed. A time-stamped description of video actions was added to each transcript so that verbal reports could be analyzed alongside students' navigation decisions within one document. You may view Camtasia recordings of think aloud sessions and a sample transcript at: http://www.newliteracies.uconn.edu/iesproject/videos/.
Missing Data
A total of 159 think aloud sessions were conducted (53 students x 3 sessions each). Seventeen sessions (approximately 10%) were not used in this analysis because of technical or other problems. These recording failures were distributed relatively evenly throughout the three sessions. (Typically, the audio recording quality was insufficient to be usable.) A final total of 142 sessions were used in our analysis. Forty students had complete audio and video recordings for all three sessions.
Analytic Procedures
Following the completion of the think aloud sessions, 142 transcripts were developed from the Camtasia video recordings. These included the transcription of the full audio recording as well as descriptions of all on-screen movements made by each informant as they read online. Then, all think aloud commentaries was parsed into idea units (Anderson, Reynolds, Schallertt, & Goetz, 1977) for analysis. An example appears at: http://www.newliteracies.uconn.edu/iesproject/videos/
The analysis of transcriptions was completed following both constant-comparative (Bogdan & Biklen, 2003; Merriam, 1988) and abductive (Onwuegbuzie & Leech, 2006) methods. Abductive coding methods (Onwuegbuzie & Leech, 2006) employ both inductive and deductive coding procedures. Initial codes were deduced from the theory base of online reading comprehension developed by Leu, Kinzer, Coiro, & Cammack (2004) which included developing questions, locating information, critically evaluating the usefulness of information, synthesizing information, and communicating information; these categories were inserted initially as primary nodes in the developing coding system. The "branches" that emerged from these primary nodes were derived through inductive procedures and constant comparative analysis, involving several of the researchers who had both administered verbal protocols and transcribed the data. The data were parsed into strategy units and inserted under the primary categories. In the process, ancillary nodes were also developed. The researchers then discussed the various elements of each strategy – those that clearly extended the existing codes as well as those that did not. New categories and extensions of categories grew from these discussions until the need to extend or revise primary nodes and all branches was exhausted from the initial set of transcripts. The codes were then inserted into the NVIVO-7 project file with distinctive descriptions that were agreed upon by the researchers. For example, under the general category of locating information, strategies for reading search engine results were categorized as follows:
1. Click and Look: Action based on proceeding systematically through search engine results (i.e. begin with first link and progress list wise, one link after the other).
2. Description Reading: Action based on specific reading of search results (i.e., identifies bolded words from keyword input, related words, etc)
3. Touring Results Page: Action based on scrolling through results page prior to close reading or change of keywords (i.e., a virtual text walk).
4. URL Reading: Action based on specific reading of URL (i.e., identifying certain elements of URL such as .com, .edu., .gov, etc).
It should also be noted that this analytic process was also informed by work taking place concurrently on a number of other independent research studies including ones by Castek (2007), Coiro (2006), Coiro & Dobler (2007), and Henry (2006) among others. Insights from these studies have informed work on this project in important ways.
As the coding of the verbal protocols continues, opportunities to revisit and revise the developing taxonomy continue to be presented and the need to reconsider descriptions of categories may again arise. This process is being carried out through discussions by researchers until agreement is reached and categories and descriptions are revised. This approach thus serves to ground the taxonomy in existing and developing theories of Internet reading comprehension while allowing for new information to emerge from the data.
Examples of Online Reading Comprehension Strategies
In this section, we present examples of two strategy categories that appear to be especially important for successful online reading comprehension: locating information and critically evaluating the usefulness of information. While our analysis is ongoing, it is becoming clear that online reading comprehension skills in these two areas may be “circuit breaker” skills; having them leads to successful online reading comprehension, while lacking them often leads to failure. There may be a useful analogy to decoding skills during offline reading comprehension, where it is difficult to successfully comprehend offline text without adequate decoding skill. Online, it appears equally difficult to successfully comprehend information if one does not possess adequate skills to locate it or if there is a failure to critically evaluate information. Sometimes the two skills interact, as when a reader must critically evaluate search engine results to determine the most useful link for a particular information purpose. At times like this, it may be especially important for successful online reading comprehension to posses both skill sets.
An Example Of A Student Using Successful Strategies For Locating Information
Heather (a pseudonym) is an example of a student who is decidedly skilled at locating information. In the interview portion of Session 3, Heather indicated that she knows how skilled she is at locating information with a search engine. She also describes knowing two online reading comprehension skills related to locating information that others might not -- selecting appropriate key words and carefully reading the “small part” (i.e., the summary) under each item that is listed in a set of search engine results:
R: What do you think you know about reading on the Internet that some kids might not know?
S: (long pause) … being able to pick out key words. Some students aren’t very good at picking out topics or whatever. Or being about to read the small part underneath it and determining if that’s what you’re looking for or not. To be able to read the summary.
Heather has recently been diagnosed with a disease called CMV. Because the third session was a student-directed reading task, Heather decided to read online about CMV, in order to learn exactly what this disease is and how one contracts it.
When it is time to begin reading online, Heather indicated that she was familiar with search engines and went immediately to Google:
S: Now, I like to use google because I’m just familiar with it.
[14:00 Opens new IE page. Goes to google by typing URL in address bar.]
Heather also showed us that she knows how to use key words and phrases to locate information:
[14:23 Types in “causes of CMV” in Google search bar.]
S: I’m going to look up the causes of CMS because that’s one of the questions.
[14:35 Opens first result in list of search engine results. ]
S: I’m going to read and see what it says.
Heather also showed us how she carefully analyzes the information that she encounters at this first location, monitoring what she reads and comparing it to her needs:
[15:03 Begins scrolling down to read the information at the first page she opens.]
S: I’m just kind of scanning it right now. I’m trying to see if it tells what causes it. It tells what it’s related to. I don’t really see much about what it is. Okay, I’m not seeing the question I was looking for.
[15:57 Hits the back button to return to search results.]
S: I’m going to go back and read some more of these and see what it says.
She also showed us that it is useful to actually read the summaries below each set of search engine results. Heather did not use the strategy often used by less successful online readers in our sample who simply use a “click and look” strategy, working one’s way down the list of results, clicking and looking quickly at each one without ever reading the summary:
[16:12 Is reading search results this time, not just clicking to see if the page has what she wants.]
R: Okay. As you’re reading here, what are you thinking? What are you looking for?
S: I’m looking for some of the main causes for it, but I’m not really finding much about what I’m looking for.
And, finally, Heather knew how to reduce the set of key words in order to find the broader set of results she needed to locate information on how one contracts this disease:
S: So, I think what I’m going to do is I’m going to go up here and change it to just CMV. Because I’m not finding much…
[16:33 Modifies key word to “CMV”]
R: And what made you decide to do that?
S: Because when you type in the causes of CMV, it tells you what it causes in people instead of the cause of getting it.
Heather showed us four important online reading comprehension skills related to locating information online. She demonstrated: (a) an understanding of how one can use a search engine to locate information; (b) how to construct key word searches related to her question; (c) how to analyze the results summary to determine if any locations meet her information needs; and (d) how to expand a key word search when the initial results produce too narrow a set of informational resources. Heather eventually located the information she was looking for because she had a sophisticated set of online reading comprehension skills related to locating information. You may view the video of this portion of Heather's online reading at: http://www.newliteracies.uconn.edu/iesproject/videos/.
An Example Of A Student Using A Less Successful Strategy For Locating Information: The “.com Strategy.”
Tyler is an example of a student who is not as skilled at locating information online. He began the first online reading session seeking to locate the site called Save the Pacific Northwest Tree Octopus (http://zapatopi.net/treeoctopus/). Like many of our less successful online readers, Tyler did not appear to know about search engines. Instead, he uses a “.com strategy” to locate information, typing in the name of the topic he is looking for in the address window and adding a ".com" at the end:
[13:38 … highlights address in address bar, types in www.savethepacificnorthwesttreeoctopus.com in the address bar, presses enter and waits]
[14:16 the url could not be retrieved…)
Because this strategy did not produce the site, Tyler tried clicking on the full URL option provided by the browser:
[14:16 clicks the link http://www.savethepacificnorthwesttreeoctopus.com and waits]
[14:55 clicks link again, waits]
[15:14 clicks back button]
Then he continued using his .com strategy, revising the name by dropping off the word “pacific” in the address window:
[15:22 highlights address bar, types in www.savethenorthwesttreeoctopus.com, presses enter and waits]
[15:55 the url could not be retrieved, clicks the link http://www.savethenorthwestreeoctopus.com and waits]
Next, he tried another option, still using a .com strategy; he drops off the word “tree” in the address window:
[16:16 highlights the address bar, types in www.savethenorthwestoctopus.com, presses enter and waits]
[16:54 the url could not be retrieved]
S: I wonder why it’s not coming up. [long pause] [indecipherable] [long pause]
Finally, he tried yet another variation of the .com strategy, adding back the term “pacific,” but dropping the term “tree.”
[17:10 highlights the address bar, types in savethepacificnorthwestoctopus and presses enter and waits]
[18:01 MSN search page appears with nothing found resembling savethepacificnorthwestoctopus,
While Tyler's actions show purposeful strategies being used to locate information,
it appears that he did not know about search engines. Tyler also did not know, for example, how to make key word searches, using instead, a “.com strategy” to try to locate information. (One suspects he might look for information on the revolutionary war by entering www.revolutionarywar.com or information on George Washington by entering www.georgewashington.com.) Having limited ability to locate information makes it difficult to successfully complete any online reading comprehension task. This portion of Tyler's's online reading may be viewed at: http://www.newliteracies.uconn.edu/iesproject/videos/.
These two cases illustrate a contrastive aspect of our analysis. By sometimes focusing on both successful and unsuccessful online reading comprehension strategies, it is possible to provide a richer approach to understanding the full dimensions of the skill sets that are important for online reading comprehension.
The Critical Evaluation of Information – Effective And Ineffective Strategies
Our data indicate that the critical evaluation of information takes place at multiple points in time during online reading comprehension. It happens as readers evaluate the question that they have, the results listed from a search engine to decide which will be most useful, the reliability of information at a site they visit, their synthesis of information to see if it meets the needs of the task they are trying to complete, and the message they are composing to make certain it makes sense.
One form of evaluation may be especially important to consider during online reading – the critical evaluation of how reliable information might be at a web site. This skill was observed during the first session with our informants, when we asked them to gather information and evaluate the reliability of the spoof site, Save The Pacific Northwest Tree Octopus (http://zapatopi.net/treeoctopus/ ). 87.5% percent of informants with recorded data (42 out of 48) reported that they thought this site was reliable; slightly more than half of our students (27) reporting it to be “very reliable.” 6 out of 48 (12.5%) students reported that the site was unreliable and each of these had just participated in a lesson that taught this site, showing students to be suspicious of information online. Data from five students were missing.
Few of our informants showed any indication of questioning the information located at this site. And, when told that the site was not reliable, some of the students insisted that we were wrong. Then, when asked to do so, very few students could use the Internet to find information that would demonstrate the site was either reliable or unreliable. Most seemed to not question the reliability of information online during reading even though many students reported, during an interview, that one should not trust information found on the Internet.
An Example Of A Reader Who Is More Successful With The Evaluation Of Online Information.
Pat was one of the few students who was able to determine that the site, Save the Northwest Pacific Tree Octopus, did not provide reliable information. The library media specialist at this school had recently used this site as an example of how some information on the Intenet is not true. Pat read the information at the site relatively quickly, spending only about a minute:
[18:08 results page opens, clicks the first result to Save The Pacific Northwest Tree Octopus]
[18:21 page opens, scrolls up and down page]
Pat then began to post his response at the blog:
[20:35 clicks on blog from task bar, scrolls up and down page]
[20:52 clicks on a previous post]
[21:12 clicks on post a comment link]
[21:22 puts cursor in comment box, begins to type]
His post at this blog read:
“The most important info that the world needs toknow is that your website is bogus. There is know such thing as a tree octopus,they can only live in water.
What kind of a joke are you guys trying to pull. No one is going to belive your bogus requests, so stop trying. the info on the website is unreliable in all ways.”
It is important to notice in this video and think aloud that Pat did not employ any strategy to critically evaluate the reliability of this site nor did he provide evidence from the Internet to support his evaluation, he only relied upon a previous classroom experience, where he had been told that the information was false. This happened with all of the other five students who indicated that this site was not reliable. None searched for information about what other people were saying on the Internet about Save the Pacific Northwest Tree Octopus or visited a site such as snopes.com to see if it was listed as a hoax site. You may view the video of this portion of Pat's online reading at: http://www.newliteracies.uconn.edu/iesproject/videos/.
An Example Of A Reader Who Is Less Successful With The Evaluation Of Online Information.
An example of a student who failed to evaluate the reliability of information presented at Save The Pacific Northwest Tree Octopus was a student we shall call Ali. Rather than initially thinking about the source and evaluating the reliability of information at this site, Ali’s first thought is about how much information is available at this site and that she is going to be able to quickly copy and paste some of that information to complete the task.
R: So tell us about what you’re thinking.
[26:00] Scrolls down and up again
S: I’m thinking there’s a lot of information
and, um, that I’m gonna copy some of it and paste it onto document Microsoft.
[26:23] Highlights and copies text
I think I’m gonna copy “The Pacific Northwest tree octopus (Octopus paxarbolis) can be found in the temperate rainforests of the Olympic Peninsula on the west coast of North America.” Copy and I’m gonna paste
[26:34] Maximizes document from “Start” bar
[26:35] Pastes onto document
The information that Ali eventually copies and pastes into the blog entry, is extensive and challenging for most seventh graders to understand. Only the final line, where she evaluates the site and indicates that she finds it reliable, is in her own words:
From: Anonymous [anonymous-comment@blogger.com]
To:* XXXXXX
Subject: [Mrs. Gonzales' Class]
paxarbolis) can be found in the temperate rainforests of the Olympic Peninsula on the west coast of North America. These solitary cephalopods reach an average size (measured from arm-tip to mantle-tip,) of 30The Cephalopods ("head-foot") are the mollusk class Cephalopoda characterized by bilateral body symmetry, a prominent head, and a modification of the mollusk foot, a muscular hydrostat, into the form of arms or tentacles. Teuthology, a branch of malacology, is the study of cephalopods.-33 cm. Reaching out with one of her eight arms, each covered in sensitive suckers, a tree octopus might grab a branch to pull herself along in a form of locomotion called tentaculation; … The history of the tree octopus trade is a sad one. Their voracious appetite for bird plumes having exhausted all the worthy species of that family, the fashionistas moved on to cephalopodic accoutrements during the early 20th Century. Tree octopuses became prized by the fashion industry as ornamental decorations for hats, leading greedy trappers to wipe out whole populations to feed the vanity of the fashionable rich. … its effects still reverberate today as these millinery deprivations brought tree octopus numbers below the critical point where even minor environmental change could cause disaster
I think that this website save the pacific Northwest Tree octapus is very relible and that it gives a lot of information and I would use it again if i am looking fro an endagered Species
Then, when told that the information at this site was unreliable, Ali told the researcher that that the researcher was wrong. She remained convinced it was reliable, even after being told that it was not:
R: You, um, what if I told you that this site was not at all reliable and that the information was not true.
S: I would say that you were wrong and that maybe you used a different a website and it’s just called the same thing because the stuff I found out was everything I needed to find out and some other stuff that I didn’t need to know so I think it’s very reliable and I disagree with you.
Summary
As is evident from these examples, our seventh-grade informants demonstrated a variety of skills and strategies in these two areas, ranging from poor to very effective. It appears that the evaluation of the reliability of information may be a particular concern, since so few students thought critically about the information that they read on the Internet. As we use these examples to refine our analytic coding scheme, we begin to think about how to apply what we have learned to help less skilled readers and interpret our analytic codes into more interpretable skill sets that can be used for assessment and instruction.
An Evolving Preliminary Taxonomy
The first step in the process we are following is the development of a preliminary coding scheme. Our analysis of think aloud data from our informants has led us to generate a coding scheme that captures the general nature of the skills and strategies our informants use during online reading comprehension. This analytic scheme is presented in Figure 1. It has been helpful in beginning to understand the large set of data that we have in our transcriptions and videos of online reading. It continues to undergo refinement as we evaluate its utility to capture all aspects of our think-aloud data.
------------------
Figure 1 Here
------------------
This work has also been helping us understand that our coding scheme, by itself may not be maximally useful to inform practice. As we begin to move from our verbal protocol analysis to thinking about how these patterns might inform our Internet Reciprocal Teaching lessons in the classroom, we thought it might be helpful to organize our observations into a preliminary, more teacher-friendly, list of skills. As a result, we have also been developing a parallel taxonomy to inform classroom instruction and assessment. We have tried to turn what we have been learning, in terms of the patterns of more successful and less successful strategy use, into action statements that might come from a highly skilled online reader. (e.g., “I know how to…”). We believe that statements like these, derived from the patterns in our data, are more likely to be helpful to classroom teachers and to inform the design of assessment instruments. Our developing classroom instructional taxonomy appears in Figure 2. This also continues to undergo refinement as the coding scheme changes.
------------------
Figure 2 Here
------------------
Discussion Work on the TICA project is ongoing, adding an important dimension to our understanding of online reading comprehension. It is providing a preliminary set of online reading comprehension skills and strategies used by proficient students within the population we are studying, seventh graders from 12 diverse and economically challenged school districts in South Carolina and Connecticut. Previous work (Coiro &
Dobler, 2007; Leu, et. al, 2005) has found that online reading comprehension shares a number
of similarities with offline reading comprehension but that online
reading comprehension is also more complex and includes a number of
important differences. This paper helps us to understand the details of this complexity and these differences. In addition, the taxonomy of online reading comprehension skills developed in this study informs both Year 2 and Year 3 work on the TICA project in important ways, helping us to more carefully construct classroom lessons during our formative experiments that will, in turn, allow us to design a more tightly crafted study in Year 3.
The study takes a somewhat innovative methodological approach to think aloud protocols, seeking to capture as much useful data as possible, while also being sensitive to the potential limitations of question probes. By varying
think aloud procedures to include uninterrupted think alouds,
structurally-prompted think alouds, and video-stimulated think alouds,
we have attempted to systematically capture think aloud data that was
both unprobed and probed. This has allowed us to take advantage of the
opportunities in both approaches, while also recognizing the
limitations in each. Uninterrupted
think aloud sessions were less likely to be influenced by experimenter
probes, but they also provided us with fewer data. Structurally-prompted and video-stimulated think aloud session allowed us to gather additional
data while evaluating comprehension processes during targeted aspects
of reading
comprehension. However, we are keenly aware of the potential impact of our
probes on student reports of their thinking. We are beginning to
evaluate any potential impact by comparing data from probed and unprobed sessions to evaluate the consequences of our probes. This may provide
useful methodological insights for future work that takes advantage of the
opportunities think-aloud verbal protocols provide for the study of
online reading comprehension.
Our work also sought to include both researcher-determined and student-determined reading tasks. Spontaneous online reading comprehension typically occurs with reader-determined questions (Eagleton & Dobler, 2007; Leu, et. al., 2004). Yet, to systematically evaluate online reading comprehension, it is essential to sometimes provide the question to informants in order to evaluate particular aspects of online reading. By asking our informants to evaluate the reliability of a "spoof site," for example, we were able to determine that they seldom thought critically about information encountered online, even when they told us in interviews that they knew information on the Internet was not always true. It is important to note, however, that this result could have been prompted by the social constraints of the situation; students may not have read critically because they assumed an adult would not ask them to read something that was untrue. Our preliminary analysis of a subsequent task, where students read another "spoof site," suggests that this may not be the case. Students seldom evaluated that information as well. If the failure to read critically is sustained in additional work, this would be a particularly troubling finding. It suggests that any instruction in online reading comprehension should pay particular attention to developing healthy skepticism about the information one encounters online. It also suggests that we need to help students develop strategies for critically evaluating information that they encounter. If, as Jefferson (2005) suggests, the survival of any democracy depends upon its citizens making informed decisions at the ballot box, it is essential that schools begin to develop these critical evaluation skills in relation to information that appears online.
In addition to evaluating researcher-determined reading tasks in this study, it was important to us that we also include student-determined reading tasks. By having a data set that evaluated online reading comprehension processes in both contexts, we will be able to develop an broader understanding of online comprehension. We plan to evaluate the extent to which student-determined, online reading is different from teacher-determined reading. This will provide useful information for both methodological and practical reasons. Methodologically, this may be important for understanding the consequences of design decisions. Practically, we will be in a better position to inform instructional decisions for achieving different types of outcomes. Preliminary analysis suggests that student-determined online reading tasks resulted in generally shorter sessions. We are still trying to understand the possible reasons for this, whether this had an impact on strategy use, and the implications of these results for both the design of upcoming studies as well as recommendations for classroom instruction.
This work employed a somewhat innovative use of video and audio capture technology to record, in real time, online reading comprehension processing. The history of research in reading comprehension has been limited by having few tools that allow us to peer into what actually takes place during online reading comprehension processing. Eye movement studies (e.g., McConkie, Kerr, Reddix, & Zola, 1988; Rayner, 1998), miscue analysis (Goodman & Burke,l 1973; Leu, 1982), and think aloud verbal protocols (Afflerbach, 2002; Pressley & Afflerbach, 1995) have all been used previously. Each, however, suffers from important limitations. Using think alouds with video and audio capture technology permits us to capture the complexities of online reading comprehension in real time with a continuous picture of what students are reading, which links they follow, and their verbal, think aloud reports. Video recordings allow us to repeatedly observe and compare selected cases, looking for patterns that will help us to better understand the nature of online reading comprehension. We believe this use of video and audio capture technology will open important new avenues for research and provide direction for future studies in this area.
The analysis of online reading comprehension strategies we are conducting provides greater detail to an emerging picture of how online reading comprehension takes place. The coding categories that are emerging provide us with a more specific description of the skills and strategies that are useful for online reading comprehension. By developing a parallel
taxonomy, based around action statements that a proficient online reader might be able to make (e.g., “I know how to…”), we also hope to make our work interpretable to classroom teachers and to those who are designing measures of online reading comprehension. New assessments are beginning to appear in this area. The ICT Literacy Assessment (Educational testing Services, 2002), the Key Stage 3 Test in ICT (National Assessment Agency, n.d.), and the 2009 PISA International Assessment of Reading (OECD, 2006) are three important new assessments that include, or will shortly include, various aspects of online reading comprehension. Each suffers, however, from the lack of a clear conceptual framework to inform the design of the assessment, largely because data are sorely lacking about the important skills required to successfully read and comprehend information on the Internet (Leu, 2007). This study may provide useful direction for the important assessment decisions that lie ahead.
Finally, this work may permit us to more thoughtfully focus attention on those students who need our support the most, students in economically challenged school districts who have access to online reading experiences the least. In any society that professes egalitarian ideals, schools should serve to provide equitable learning experiences for all of its children. The influence of recent public policies in the U.S. have recently, and largely unknowingly, encouraged our economically challenged schools to ignore the types of online reading experiences that will be central to success in the 21st century. Unfortunately, reading instruction in these schools is focused largely on helping students to pass assessments that have little to do with online reading comprehension, the type of reading that is likely to be increasingly important in a globalized information society. Before we should expect that situation to change, however, policy makers will require data that illustrates the nature of the challenge and specific guidance for solving it. This work may provide an important initial step in that direction.
References
Alvermann, D. E. (2006). Technology use and needed research in youth literacies. In M. McKenna, L. Labbo, R. Kieffer, & D. Reinking (Eds.), International handbook of literacy and technology (2nd ed.). Mahwah, NJ: Erlbaum.
Afflerbach, P. (2002). The use of think-aloud protocols and verbal reports as research methodology. In M. Kamil (Ed.) Methods of literacy research, pp. 87-103 Hillsdale, NJ: Erlbaum Associates.
Anderson, R.C., Reynolds, R.E., Schallert, D.L., & Goetz, E.T. (1977). Frameworks for comprehending discourse. American Educational Research Journal, 14, 367-381.
Bogdan, R. C., & Biklen, S. K. (2003). Qualitative research for education: An introduction to theories and methods, (5th ed.). Boston: Allyn & Bacon.
Borzekowski, D. L., Fobil, J. N. & Asante, K.O. (2006). Online access by adolescents in Accra: Ghanaian teens' use of the Internet for health information. Developmental Psychology, 42, 450-458.
Castek, J. (2007). An examination of classroom instruction that integrates the new literacies of online reading comprehension: Exploring the contexts that facilitate acquisition and the learning outcomes that result. Doctoral dissertation proposal, University of Connecticut.
Center on Education Policy. (March, 2006). From the capital to the classroom: Year 4 of the No Child Left Behind Act. Center on Education Policy: Washington, D.C. Retrieved March 15, 2007 from http://www.cep-dc.org/nclb/Year4/CEP-NCLB-Report-4.pdf.
Chandler-Olcott, K., & Mahar, D. (2003). “Tech-savviness” meets multiliteracies: Exploring adolescent girls’ technology-mediated literacy practices. Reading Research Quarterly, 38, 356-385.
Coiro, J. L. (2006). Exploring changes to reading comprehension on the Internet: Paradoxes and possibilities for diverse adolescent readers. Doctoral dissertation proposal. The University of Connecticut.
Coiro, J. & Dobler, E. (2007). Exploring the online reading comprehension strategies used by sixth-grade skilled readers to search for and locate information on the Internet. Reading Research Quarterly, 42, 214-257.
Coiro, J., Knobel,M., Lankshear, M. & Leu, D.J. (Eds.). (in press). Handbook of research on new literacies. Lawrence Erlbaum, Mahwaw, NJ.
de Argaez, E. (January, 2006). Internet world stats news, 14. Retrieved December 15, 2006 from, http://www.internetworldstats.com/pr/edi014.htm#3.
Finn, J.D. (1989). Withdrawing from school. Review of Educational Research, 59, 117-142.
Finn, J.D. (1993). School engagement and students at risk. Washington, DC: U.S. Department of Education, National Center for Education Statistics [On-line]. Available at: http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=93470.
Fleiss, J.L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378-382.
Friedman, T. L. (2005). The world is flat: A brief history of the twenty-first century. New York, NY: Farrr, Straus, and Giroux.
Goodman, K & Burke, C. (1973). Theoretically based studies of patterns of miscues in oral reading performance, final report. Wayne State University, Detroit. (Eric Document Reproduction Service No. ED 179 708)
Henry, L.A. (2006). Exploring new literacies pedagogy and online reading comprehension among adolescents and their teachers: Issues of social equity or social exclusion? Doctoral dissertation proposal. The University of Connecticut. Available at: http://newliteracies.uconn.edu/lahenry/proposal_HENRY.pdf
Henry, L. A., Mills, C., Rogers, A., and Witte, J. (2006). In D. Reinking and Leu, D.J. (Chairs). Developing Internet reading comprehension strategies among adolescents at risk to become dropouts. An online survey of Internet use among adolescents at risk of dropping out of school. Poster paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
International Reading Association (IRA). (2002). Integrating literacy and technology in the curriculum: A position statement. Newark, DE: International Reading Association.
Internet World Stats (2006). Internet Usage Statistics – The Big Picture: World Internet Users and Population Stats. Retrieved from: http://www.internetworldstats.com/stats.htm
Jefferson, T. (2005). Autobiography of Thomas Jefferson. New York: Dover.
Labbo, L.D., & Reinking, D. (1999). Multiple realities of technology in literacy research and instruction. Reading Research Quarterly, 34, 478-492.
Lankshear, C., & Knobel, M. (2003). New literacies: Changing knowledge and classroom learning. Milton Keynes: Open University Press.
Lebo, H. (2003). The UCLA Internet report: Surveying the digital future, year three. Los Angeles: UCLA Center for Communication Policy. Retrieved March 26, 2007 from http://www.digitalcenter.org/pdf/InternetReportYearThree.pdf
Leu, D. J., Jr. (1982). Oral reading error analysis: A critical review of research and application. Reading Research Quarterly, 17, 420-437.
Leu, D. J. (January, 2007) Expanding the reading literacy framework of PISA 2009 to include online reading comprehension. A working paper commissioned by the PISA 2009 Reading Expert Group.
Leu, D. J., Ataya, R., & Coiro, J. (December, 2002). Assessing assessment strategies among the 50 states: Evaluating the literacies of our past or our future? Paper presented at the National Reading Conference. Miami, FL.
Leu, D.J., Castek, J., Hartman, D., Coiro, J., Henry, L.A., Kulikowich, J. & Lyver, S. (2005). Evaluating The Development of Scientific Knowledge and New Forms of Reading Comprehension During Online Learning. Final report submitted to the North Central Regional Educational Laboratory/Learning Point Associates.
Leu. D. J., Jr., Kinzer, C.K., Coiro, J., & Cammack, D. (2004). Towards a theory of new literacies emerging from the Internet and other ICT. In R.B. Ruddell & N. Unrau (Eds.), Theoretical Models and Processes of Reading, 5th Edition, 1570-1613.
Leu, D. J. & Reinking, D. (2005). Developing Internet Comprehension Strategies Among Adolescent Students At Risk to Become Dropouts. U. S. Department of Education, Institute of Education Sciences Research Grant. Retrieved June 20, 2006 at http://www.newliteracies.uconn.edu/ies.html.
Leu, D. J., Zawilinski, L., Castek, J., Banerjee, M., Housand, B., Liu, Y., and O’Neil. M (in press). What is new about the new literacies of online reading comprehension? In A. Berger, L. Rush, & J. Eakle (Eds.). Secondary school reading and writing: What research reveals for classroom practices. National Council of Teachers of English/National Conference of Research on Language and Literacy: Chicago, IL.
Livingstone, S. and Bober, M. (2005) UK children go online: Final report of key project findings. London: LSE Report, April 2005. Retrieved March 28, 2007 from www.children-go-online.net.
McConkie, G.W., Kerr, P.W., Reddix, M.D. and Zola, D. (1988). Eye movement control during reading: The location of initial eye fixations. Vision Research, 28, 1107-1118.
Merriam, S.B. (1988). Case study research in education: A qualitative approach. San Francisco: Jossey-Bass.
National Center for Education Statistics (2003). Dropout rates in the United States 1998. Washington, DC: U.S. Department of Education.
Morgan, D. L. (2007). Paradigms Lost and Pragmatism Regained: Methodological Implications of Combining Qualitative and Quantitative Methods. Journal of Mixed Methods Research, 1, 48-76.
National Assessment Agency (n.d.). The essential guide to the key stage 3 ICT Test. Retrieved from http://www.naa.org.uk/naaks3/ on March 20, 2007.
New London Group. (1996). A pedagogy of multiliteracies: Designing social futures. Harvard Educational Review, 66(1), 60-92.
No Child Left Behind Act of 2001, Pub. L. No. 107-110, 115 Stat. 1425 (2002). Retrieved December 10, 2003, from http://www.ed.gov/policy/elsec/leg/esea02/index.html
OECD (2006). PISA - The OECD programme for international student assessment. Retrieved March 25, 2007 from http://www.oecd.org/dataoecd/51/27/37474503.pdf
Onwuegbuzie, A. & Leech, N. L. (October, 2006). Qualitative data analysis: A step-by step approach. Presentation provided by Education Technology Services of Clemson University. Clemson, SC.
Pew Internet & American Life Project (2005). Teens and Technology. Retrieved April 15, 2006 from http://www.pewinternet.org/topics.asp?c=4.
Pavio, A. (1990). Mental representations: A dual-coding approach. New York: Oxford University Press.
Palincsar, A.S. & Brown, A.L. (1984). Reciprocal teaching of comprehension-fostering and comprehension-monitoring activities. Cognition and Instruction, 1, 117-175.
Palincsar, A.S. (1986). Reciprocal teaching. In Teaching reading as thinking. Oak Brook, IL: North Central Regional Educational Laboratory.
Pressley, M., & Afflerbach, P. (1995). Verbal protocols of reading: The nature of constructively responsive reading. Hillsdale, NJ: Lawrence Earlbaum.
RAND Reading Study Group. (2002). Reading for understanding: Toward an R&D program in reading comprehension. Santa Monica, CA: Rand.
K. Rayner. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372-422.
Reinking, D. (1997). ME and my hypertext:) A multiple digression analysis of technology and literacy (sic). The Reading Teacher, 50, 626–643.
Reinking, D. (2001). Multimedia and engaged reading in a digital world. In Ludo Verhoeven and Catherine Snow (Eds.), Literacy and Motivation. Mahwah, NJ: Lawrence Erlbaum.
Reinking, D., & Bradley, B. A. (2004). Connecting research and practice using formative and design experiments. In N. K. Duke & M. H. Mallette (Eds.), Literacy research methodologies (pp. 149-169). New York: Guilford.
Spiro, R. & Jehng, J.C. (1990). Cognitive flexibility and hypertext: Theory and technology for the nonlinear and multidimensional traversal of complex subject matter. In D. Nix & R. Spiro (Eds.), Cognition, education and multimedia: Exploring ideas in high technology. Hillsdale, NJ: Erlbaum.
U. S. Department of Commerce: National Telecommunications and Information Administration (2002). A nation online: How Americans are expanding their use of the Internet. Washington, DC: U.S. Department of Commerce.
Wylie, V.L., & Hunter, W.A. (1994). The dropout problem: Can schools meet the challenge? NASSP Bulletin, 78, 74-80.
Table 1.
A comparison of mean scores on the selection criteria used to identify student informants for the think aloud sessions.
Total Sample Informants
(n=1,025) (n=53)
Range Mean SD Range Mean SD
Online Reading
Comprehension Ability 0-27 12.14 5.15 17-27 20.55 2.40
(Total = 32)
Frequency of Internet
Use (Total=270) 0-178 57.56 34.93 70-176 103.72 25.97
Diversity of Internet Use 0-50 23.43 9.16 26-50 34.59 6.77
(Total = 54)
Expertise in Explaining
Strategies 0-2 0.71 0.70 1-2 1.36 0.48
(Total = 2)
Comfort in Working
With an Adult 0-3 2.04 0.91 1-3 2.42 0.64
and Thinking Aloud
(Total - 3)
Figure Captions
Figure 1. A preliminary coding scheme of online reading comprehension strategies.
Figure 2. What do good online readers know? A preliminary taxonomy of skills and strategies for instruction and evaluation
Figure 1.
I. Question or problem: Identifying a question or defining a problem
A. Understanding the Audience: Student makes specific reference to audience or personal needs in relation to identifying or developing questions.
B. Developing a Question, Problem or Topic: Includes narrowing, expanding and/or refining questions.
C. Shift in Question, Problem or Topic: Students changes question or problem based on the availability of information or presentation of new information (abandonment of original question).
D. Understanding the Question: Strategies for checking or understanding the question (rereading task/ self-monitoring).
II. Locate: Using the Internet to locate an information resource (using a search engine and/or other methods).
A. Keywords: Keyword entry strategies: enters a keyword or words to search in a search engine.
1. Spelling: Strategies for obtaining the correct spelling of keywords.
B. Reading Results: Strategies for reading and selecting an information source on a page of search engine results.
1. Click and Look: Action based on proceeding systematically through search engine results (i.e. begin with first link and progress listwise, one link after the other).
2. Description Reading: Action based on specific reading of search results (i.e. identifies bolded words from keyword input, related words, etc)
3. Touring Page: Action based on scrolling through results page prior to close reading or change of keywords (i.e., a virtual text walk).
4. URL Reading: Action based on specific reading of URL (i.e., identifying certain elements of URL such as .com, .edu., .gov, etc)
C. Search: Methods and strategies for searching for information on the Internet
1. Engine: Using a search engine to locate an information resource (includes strategies or reasons for choosing a search engine).
2. Dotcom Strategy: typing in the name or topic in the address bar and adding .com to the end.
3. Other: Using other methods (not search engine) to locate an information resource
D. Webpage Reading: Reading strategies used at a webpage to locate information or decide where to go next (not reading search engine results).
1. Design Elements: Reading of visual features and attributes of webpage to target information (i.e., menu options, links, headings, images, etc.)
2. Summarize: Verbal summarization of information during webpage reading.
3. Text Walk: Actions based on quick review of webpage (similar to touring of results page) prior to close reading.
III. Evaluate: General statements or actions about critically evaluating information.
A. Accuracy: Evaluating information based on the degree to which it is correct; the extent to which information contains factual and updated details that can be verified by consulting alternative and/or primary sources.
1. Accuracy Confirmation: Student confirms or rejects information from a secondary source.
2. Accuracy Shift in Reading: Student changes information source based on the accuracy of information.
B. Bias and Stance: Evaluating information in relation to the stance an author takes (the lenses, viewpoint and/or agenda embedded within the information)
1. Bias-Stance Confirmation: Student confirms or rejects information source based on the bias or stance of a secondary source.
2. Bias-Stance Shift in Reading: Student changes information source based on the bias or stance of the author or sponsor of webpage.
C. Relevancy: Evaluating information in relation to its utility/relevancy to the question or problem: the information's level of importance to a particular reading purpose or stated information need.
1. Relevancy Confirmation: Student confirms or rejects information source based on its level of relevancy
2. Relevancy Shift in Reading: Student changes information source based on it level of relevancy
D. Reliability: Verification of information for reliability: the information's level of trustworthiness based on information about the author and the publishing body
1. Reliability Confirmation: Student confirms or rejects information source based on the reliability of the information
2. Reliability Shift in Reading: Student changes information source based on it level of reliability
IV. Synthesize: Integrating information from multiple resources.
A. Combination of Text and Multimedia: Synthesizing information across both text-based and visual sources of information (a combination of the above elements).
B. Multimedia-based: Synthesizing information using multiple visual/audio sources (i.e. visual images, charts, diagrams, videos, audios, etc)
C. Text-based: Synthesizing information across multiple text sources.
V. Communicate: Use of one or more of the designated ICTs to share a response.
A. Audience and Purpose: Skills and strategies used that relate to the purpose of communicating information
1. Audience (Content): Monitoring communication of information for audience or voice (i.e., formal versus informal writing styles)
2. Tool Selection (Form): Identification of why and how students choose a particular tool for the communication of information
B. Formatting: Selection of particular fonts, colors, adding clip art, inserting images into document communicated via ICTs
C. Note taking: Identification of skills and/or strategies used for note taking purposes
1. Hard: Paper/pencil based note taking.
2. Soft: Electronic/digital note taking.
D. Proofreading: Strategies used for revising and editing a written response.
E. Technical Aspects: Identification of technical aspects related to using ICTs (i.e., interfaces, etc)
1. Blog: Skills related to posting a comment to a blog
2. Email: Skills related to sending an email message, including attachments
3. Instant Messenger: Skills related to using instant messenger
VI. Other Strategies: Strategies outside the theoretical framework
A. Instruction: Statements related to instruction for using the Internet
B. Technical Skill: Technical skills that show specific knowledge about the computer and or software interface (not related to communication tools as identified in other nodes; i.e. use of back button, copy/past function/ find feature, etc.)
Figure 2.
Asking Questions/Solving Information Problems
1. I know what a really good question is.
2. I know who my audience is and what their needs are.
3. I know that revising the question, when I get new information, often makes it better.
4. I know how to use the Internet to develop background knowledge and make my question better.
5. I know that I need to remember my question and not get distracted.
Locating Information: Search Engines
1. I know how different search engines work.
2. I know when to change and use a different search engine.
3. I know simple strategies for making my search more specific.
4. I know advanced search strategies and when they could be useful.
5. I know how to search generally for useful key words when I do not know much about the topic.
6. I know how to use quotation marks in my search terms and when these are most useful.
7. I know how to read search engine results and usually do so.
8. I know how to evaluate possible search engine results.
9. I know how to search for images as well as text.
Locating Information: Web Pages
1. I know the typical “geography” of websites.
2. I know how to quickly skim information at a website to find the information I need, the link to take me there, or when I need to go somewhere else.
3. I can predict what kind of information will be at most links.
4. I know how to evaluate information that I find on a web site.
5. I know when I have located a useful website.
Evaluating Information
1. Understanding – I know when information makes sense to me.
2. Relevancy – I know when information meets my needs.
3. Accuracy - I know how to verify information with another source.
4. Reliability - I know how to tell when information can be trusted.
5. Bias - I know that everyone “shapes” information and how to evaluate this.
6. Stance - I am a “healthy skeptic” about online information.
Synthesizing Information
1. I know how to construct the information I need as I read selected information.
2. I know which information to ignore when I read.
3. I know how to put information together, and make inferences when it is missing, to answer my question.
4. I know when I have my answer.
5. I know how to integrate multiple media sources to derive my answer.
Communicating Information
1. I understand my audience’s needs.
2. I know how to construct a clear and unambiguous message so that the reader knows what I mean.
3. I know how NOT to make people upset with me from the way I write my message.
4. I know how/when to access and publish information on a blog.
5. I know how/when to access, revise, and publish information on a wiki.
6. I know how/when to use email.
7. I know how/when to use IM.
8. I know how/when to attach a document to my messages.
9. I know how/when to use other communication tools.
Author Notes
The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305G050154 to The University of Connecticut and Clemson University. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.