WHAT ANALYZING STUDENT EVALUATIONS CAN TELL US ABOUT SOURCE EVALUATION METHODS

Amanda Kaufman, ZSR Library, Wake Forest University

Elizabeth Ellis, ZSR Library, Wake Forest University

Dr. Alan Brown, Education Dept., Wake Forest University

Wake Forest University| 2024

SLIDEDECK

• Share the results of an on-going study we are doing comparing source evaluations from students using either the CRAAP Test and the SIFT Method
• (Spoiler: SIFT performed better, but there’s some “buts”!)
• Share pre-course results
• Share post-instruction results
• Discuss interesting findings and implications

AGENDA

THE CRAAP TEST

Currency - the timeliness of the information

Relevance - how the info meets the information need

Authority - the source of the information

Accuracy - the reliability/correctness of the information

Purpose - the reason the information exists

CRITICISMS OF CRAAP

• It is not a web-native evaluation technique.
• Heavily influenced by library collection development criteria (Caulfield 2018)
• “An analog approach” to a digital challenge (Breakstone et al. 2018)
• Too Long. Unrealistic to expect students to ask themselves 25 different questions about a source, which CRAAP does (Breakstone et al. 2018)
• Encourages binary thinking where sources are either “good,” or “bad” (Lenker 2017)

CRITICISMS OF CRAAP

• Importantly, Breakstone et al. (2018) also noted how easy it was for non-credible sources to “pass” the CRAAP Test.
• It’s a “vertical reading” technique – this refers to the idea of learning more about a source’s reliability from the source itself

• Wineburg and McGrew (2019) published their study showing that researchers using lateral reading methods performed better on tasks related to evaluating information online than those using other methods.
• Lateral reading refers to the concept of learning more about a source’s reliability from outside sources
• Mike Caulfield develops the SIFT Method for online website evaluation, in which three of the four main steps in SIFT include lateral reading techniques

CONTEXT

THE SIFT METHOD

Stop - the timeliness of the information

Investigate the Source

Find Better Coverage

Trace Claims, Quotes, and Media to Original Context

CRITICISMS OF SIFT

• Both SIFT and CRAAP are reactive frameworks, as opposed to proactive frameworks (Bull, 2021).
• Proactive evaluation asks students to “evaluate what they’re seeing but consider why they’re seeing what they do and what might be missing” (Bull, 2021).

TAKING A CLOSER LOOK

• We gave pause at what they felt were reductive conversations around the CRAAP Test and the SIFT Method.
• CRAAP = bad; SIFT = good
• There was also a feeling of “throwing the baby out with the bathwater” -- weren’t some elements of the CRAAP Test are still valuable?

​

• Like with the CRAAP Test, there weren’t a lot of independent studies corroborating the effectiveness of the SIFT Method.

TAKING A CLOSER LOOK

• We decided to test both the CRAAP test and SIFT Method
• Sample consisted of undergraduate students at Wake Forest University in Winston-Salem, NC
• Students were enrolled in LIB 100: Academic Research and Information Issues, a half-semester, 1.5 credit course
• Data was collected over the course of the 2023-24 academic year
• Participants submitted both pre-course and post-instruction source evaluations

Questions we set out to answer:

• How do college students evaluate online sources in an academic setting before learning a formal evaluation method?
• Are the CRAAP Test and the SIFT Method effective methods for source evaluation? Is one method more effective than the other?
• What are some common errors students make when evaluating sources in academic settings (both when using a formal method and not)?
• What are the teaching implications for all of the above?

RESEARCH QUESTIONS

PARTICIPANTS

​

First Years: 12 (10.2%)

Sophomores: 29 (24.6%)

Juniors: 22 (18.6%)

Seniors: 55 (46.6%)

​

Total: 118

​

Students by School Year:

PARTICIPANTS

Students in this study took LIB100:

​

Online: 97 (82.2%)

In-Person: 21 (17.8%)

​

Total: 118

Course Modality

PARTICIPANTS

​

CRAAP Group: 49

SIFT Group: 69

Total: 118

​

We plan to enroll more students in the CRAAP group in the fall.

Experimental Groups

The following homework prompt will ask you to evaluate an online source using whatever evaluation skills you have learned up to this point in your educational career. Since we have not learned anything yet in this course, this homework prompt is graded on completion alone.

​

Please provide a 1-2 paragraph evaluation of the following source:

• Michaels, Patrick J. and Ryan Maue. (2018). Thirty Years On, How Do Global Warming Predictions Stand Up? Wall Street Journal.

​

In your paragraph(s), consider whether this source should be considered reliable and trustworthy and provide a justification for this decision(s) with evidence.

PROMPT

• The source was trustworthy and credible:
• The source was unbiased:
• The author(s) were credible and reliable:
• The author(s) claims were accurate:
• Students could use this source in a paper without reservations:
• I had concerns about the trustworthiness or credibility of the source:
• I had concerns that the source was biased:
• I had concerns about the reliability of the authors from the source:
• I had concerns about the source's accuracy:
• I would caution students from using this source in a paper without looking into it further:

​

LIKERT-SCALE QUESTIONS

PRE-COURSE EVALUATION

During the first week of class, students were asked to evaluate an opinion piece in the Wall Street Journal written by Patrick J. Michaels titled, “30 Years On, How Do Global Warming Predictions Stand Up?”

• Michaels had a PhD in ecological climatology
• Michaels was a well-known climate change skeptic -- well-known enough to have an obituary in the New York Times (see photo).
• He worked for the Cato Institute and other libertarian think tanks. He received funding from the fossil fuel industry.
• The article is co-authored by another Cato Institute writer, Ryan Maue.

PRE-COURSE EVALUATION

The WSJ piece attempts to discredit the findings of James Hansen’s 1988 study on the relationship between greenhouse gasses and global warming, which was the basis for Hansen’s 1988 congressional testimony on climate change. It does so by:

• Misrepresenting Hansen’s study
• Omitting climate data from the 2015-2016 El Niño.

They end the article with this: “it’s time to acknowledge that the rapid warming he predicted isn’t happening.”

RATINGS

• Student correctly agreed that the source was not credible and gave sensible reasons
• We did not expect students to discover the authors’ ideological backgrounds
• Likert answers were consistent or mostly consistent with written response

​

Correct

• Student may have been technically correct but gave multiple contradictory or wishy-washy statements.
• Student’s reasoning was logically flawed or gave raters major pause
• Likert answers were inconsistent with written response or wishy-washy

Half-Correct

• Student was incorrect
• Likert answers were consistent with the (incorrect) written response

Incorrect

HALF-CORRECT EXAMPLES:

Wishy-Washy:

“This article is hard for me to determine as trustworthy or not... While Michaels and Maue have credentials of their own it is hard to verify their claims without further explanations.... I would not take this article as hundred percent true, but also would not say it is false and I view it as a informative reading that readers should explore more"

Opinions Are “Never” Reliable:

“... I do not think it is very reliable at all because the first thing I noticed when I clicked on the paper is that it is an opinion piece, these are never very reliable.”

PRE-COURSE EVALUATIONS

​

Correct: 27 (22.88%)

Half-Correct: 23 (19.49%)

Incorrect: 68 (57.63%)

​

Results

CORRECT RESPONSES:

INTERESTING FINDINGS:

CORRECT PRE-COURSE EVALUATIONS

1

2

3

17 of 27 (62.96%) of students who got the evaluation correct on the pre-course survey performed a lateral reading move (either Investigated the Source or Found Better Coverage, or both)

A high percentage of students who answered this question correctly relied on rhetorical analysis (a vertical reading method). Almost all students pointed to the source’s bias (92.59%), with many students pointing out emotionally charged language and tone

Students in this group indicated a preference for scholarly, peer reviewed sources and indicated the source was out of line with scientific consensus on climate change

HALF- CORRECT RESPONSES:

INTERESTING FINDINGS:

HALF-CORRECT PRE-COURSE EVALUATIONS

1

2

Students in this category were confused about the nature of opinion pieces, with many viewing them in black and white terms. Students in this category were more likely to make statements like “opinion pieces are not credible.”

Students in this category commented on the author’s credentials as a marker of reliability, as well as the Wall Street Journal’s reputation, despite recognizing that it was highly opinionated and lacked evidence

INCORRECT RESPONSES:

INTERESTING FINDINGS:

INCORRECT PRE-COURSE EVALUATIONS

1

2

3

Students in this category were more likely to trust the source because it appeared in a mainstream publication (The Wall Street Journal)

Students in this category were more likely to trust the source because of the author(s) credentials

Many students in this category acknowledged that the source was biased. Bias and credibility do not preclude one another, but in this case the students misjudged the source’s level of bias.

4

Students in this category demonstrated a lower level of information literacy, as evidenced by mentioning superficial elements like the presence of a copyright logo, and evaluating the wrong author

POST-INTERVENTION EVALUATION

After being taught either CRAAP or SIFT, students were asked to evaluate a web article called “The Masking Debate is Settled,” by Robert Malone.

• Malone has BS in biochemistry and is a MD
• He was involved with early research on mRNA vaccine development, but the extent is disputed
• Malone was called “a COVID Misinformation Star” by the NY Times, promoting hydroxychloroquine and ivermectin for treatment of Covid-19
• He has taught at various institutions, founded Atheric Pharmaceutical in 2016, and was chief medical officer at Alchem Laboratories

POST-INSTRUCTION EVALUATION

Malone’s piece appeared on the Brownstone Institute’s website, which Wikipedia describes as “a think tank that opposes various measures against COVID-19, including masking and vaccine mandates,” founded in 2021 by Jeffrey Tucker, a libertarian entrepreneur.

​

The brief article uses inflammatory images and language, article headline screenshots about mask mandates, and selective fact picking from a Cochrane Library Review about the efficacy of masking in reducing the spread of respiratory illnesses.

​

POST-INTERVENTION RESULTS

Students who used the SIFT Method performed better on the source evaluation than students who used the CRAAP Test.

​

65% of students using SIFT Method were correct vs. 39% of CRAAP Test Students.

​

​

Results

CRAAP TEST RESULTS

Correct:

19 (38.78%)

Half-Correct:

14 (28.57%)

Incorrect:

16 (32.65%)

​

Total: 49

CRAAP Results

CORRECT CRAAP RESPONSES:

INTERESTING FINDINGS:

CORRECT CRAAP EVALUATIONS

1

2

Many (63%~) of the students who answers the evaluation correctly “Investigated the Source,” a lateral reading that is not explicitly laid out in the CRAAP Test. This move which had them learning more about the authors’ reliability on Google or Wikipedia.

Students were also tipped off by the biases in the source, relying on rhetorical clues like language and the “burning mask” photo

3

Many students (47%~) questioned the credibility of the authors’ sources, like the DailyMail, etc...

HALF-CORRECT CRAAP RESPONSES:

INTERESTING FINDINGS:

HALF-CORRECT CRAAP EVALUATIONS

1

2

Students in this group were likely to recognize the source’s bias, but many thought the authors’ educational credentials meant that the source was still credible.

Students in this category were less likely to see problems with the evidence used. Many pointed to the article’s outside sources as evidence pointing towards its credibility.

INCORRECT CRAAP RESPONSES:

INTERESTING FINDINGS:

INCORRECT CRAAP EVALUATIONS

1

Among Correct, Half-Correct, and Incorrect responses, the Incorrect group’s top responses were more likely to feature all steps of the CRAAP Test.

2

This group relied heavily on the authors’ educational credentials and the use of outside sources as justifications for the source’s credibility.

​

“References are provided at the end of each section, showing that the authors are not just making stuff up.”

​

SIFT METHOD RESULTS

Correct:

45 (65.22%)

Half-Correct:

12 (17.39%)

Incorrect:

12 (17.39%)

​

Total: 69

SIFT Results

CORRECT SIFT RESPONSES:

INTERESTING FINDINGS:

CORRECT SIFT EVALUATIONS

1

2

3

Most students in this group were able to correctly determine the author/publishers lack of credibility using the Investigate the Source step (this is the first lateral reading move!) .

42%~ (n=19) of students in this group did not complete all four steps, with “Trace Claims, Quotes, and Media....” being the most dropped step -- it’s also the final step.

64%~ (n = 29) of students in this group called out the authors for misconstruing or misrepresenting evidence. They made this determine by either tracing claims or through rhetorical analysis.

4

This group was more likely to mention this article not being inline with scientific consensus (n = 14, 31%~)

HALF-CORRECT SIFT RESPONSES:

INTERESTING FINDINGS:

HALF-CORRECT SIFT EVALUATIONS

1

2

Many students in this group did not perform the lateral reading steps well. For example, one student found the Wikipedia article for the author, but then used Wikipedia’s quote from the Brownstone Institute’s About Page. Others seemed to be relying on information from Google’s featured snippets/extracts.

Several students in this category did not seem to want to take the time to correctly perform the later lateral reading moves. They might make passing reference to tracing claims, but not actually do it.

Find Better Coverage: “To get a well-rounded view, it's good to check other sources too, like news websites or health organizations, to see what they say about masks and COVID-19 safety measures.”

INCORRECT SIFT EVALUATIONS:

INTERESTING FINDINGS:

INCORRECT SIFT EVALUATIONS

1

2

3

Some students in this performed lateral reading searches using Google, but Google gave them results that agreed with the Brownstone Institute. We were suspicious that students’ already calibrated Google algorithms might prevent them from getting accurate lateral reading results.

91.66% (n=11) of students in this group did not complete the steps of the SIFT method in their evaluations, with Trace Claims being the most dropped step. In fact, 56.5%~ of all SIFT students did not complete all the steps, which was much higher than the CRAAP group. (We need to re-code this, but the CRAAP group was between 10-20% non-completion).

Like with the half-correct responses, students in this group seemed to struggle to correctly perform the lateral reading steps.

Lateral Reading Works; But Advanced Techniques Require Time

• Students who laterally read tended to perform better on the evaluations than students who did not.
• Some students struggled with performing the lateral reading steps correctly or rushed through these steps.
• Teaching mastery of these skills may require more time than one information literacy sessions.
• We also speculate SIFT steps require more effort than CRAAP, which may impact how likely students are to complete the method.
• We debated recommending teaching just “Investigating the Source” if you have limited instruction time, but had concerns that students’ algorithms might too heavily influence their search results.

OVERALL TEACHING IMPLICATIONS

Binary Thinking Continues to Be a Problem

• Students in this age group are still moving through Perry’s Stages of Cognitive Development (dualism --> commitment in relativism) which was demonstrated in a lot of their reasoning, which spoke of types of sources as “good” or “bad” in binary terms
• We may accidentally promote binary ways of thinking about sources
• Librarians should emphasize the contextual nature of credibility when it comes to opinion, scholarly articles, etc...

OVERALL TEACHING IMPLICATIONS

Advocating for Including Lateral Reading into Curriculums

• Many of our students came into the class with excellent rhetorical analysis skills (e.g. were able to identify bias by analyzing claims, tony and language, etc...) and were already heavily attuned to the differences between opinion and fact.
• These are vertical reading skills they are being taught effectively somewhere (K-12?)
• Librarians could play a role in advocating for the inclusion of lateral reading skill sets into K-12 and college curriculums

OVERALL TEACHING IMPLICATIONS

Source Evaluation Methods Contextual; Not One Size Fits All

• SIFT and other lateral reading methods seem to work well in the context of evaluating “everyday” online information (e.g. a link a friend shares on social media), but may not work perfectly in academic contexts (e.g. evaluating a list of references generated by an AI tool or a library database)
• Some vertical reading techniques described and questions asked in the CRAAP Test are still important to learn.

(Tools, Tests, and Checklists: The Evolution and Future of Source Evaluation Frameworks by Sye and Thompson, 2023).

OVERALL TEACHING IMPLICATIONS

NEXT STEPS

We plan to add more participants to the CRAAP group this fall through in-person and online groups.

More CRAAP participants

Clarify for participants that the source being evaluated is from the context of a research project.

Further contextualize use of source

QUESTIONS?

References

Breakstone, J., McGrew, S., Smith, M., Ortega, T., & Wineburg, S. (2018). Why we need a new approach to teaching digital literacy. Phi Delta Kappan, 99(6), 27–32.

https://doi.org/10.1177/0031721718762419

Bull, A., MacMillan, M., & Head, A. (2021). Dismantling the evaluation framework. In the Library with the Leadpipe.

www.inthelibrarywiththeleadpipe.org/2021/dismantling-evaluation/

Caulfield, M. (2018). A short history of CRAAP. Hapgood. https://hapgood.us/2018/09/14/a-short-history-of-craap/

Lenker, M. (2017). Developmentalism: Learning as the basis for evaluating information. Portal: Libraries and the Academy, 17(4), 721–737.

https://doi.org/10.1353/pla.2017.0043

Sye, David, and Dana Thompson. “Tools, Tests, and Checklists: The Evolution and Future of Source Evaluation Frameworks.” Journal of New Librarianship 8, no. 1 (March 8,

2023): 76–100. https://doi.org/10.33011/newlibs/13/9.

Wineburg, S., & McGrew, S. (2019). Lateral reading and the nature of expertise: Reading less and learning more when evaluating digital information. Teachers College Record:

The Voice of Scholarship in Education, 121(11), 1–40. https://doi.org/10.1177/016146811912101102

​