Cut the CRAAP: Teaching Information Evaluation in the Misinformation Age

Kara Blizzard, Librarian, University of Alberta

Presentation Outline

Abstract

The checklist approach

The prevalence of checklists

Criticisms of the checklist

Alternatives to the checklist

Barriers to using critical approaches

Misinformation and disinformation

Changing the broader approach

Reflection questions

References

2

Abstract

The COVID-19 pandemic has shown the harmful effects of misinformation and disinformation, from issues around mask-wearing and physical distancing to vaccine hesitancy. Many academic librarians teach library users how to evaluate information. Checklists such as the CRAAP test continue to be central to this teaching, despite substantial evidence of the limitations and even potential harm of checklist approaches. This poster explores the prevalence of checklists in teaching information evaluation, key criticisms of these approaches, and some of the alternatives that exist. It also offers questions for reflection on our teaching practices in the context of current events.

3

The Checklist Approach

Historically, librarians used checklists to assess materials for inclusion in print collections (Caulfield, 2018). As the internet developed, they adapted these checklists to teach students to evaluate information (Meola, 2004). The CRAAP test has become the most common acronym for teaching information evaluation. It stands for Currency, Relevance, Authority, Accuracy, and Purpose (Blakeslee, 2004); Figure 1 shows an excerpt from a worksheet that uses the CRAAP test. There are also many variations that use different acronyms but similar criteria.

4

Figure 1. Excerpt from CRAAP Test Worksheet by Milner Library, Illinois State University, licensed under CC-BY 4.0.

Does the source pass the CRAAP test?

Yes must be selected for each component to pass the CRAAP test.

Currency Publication date is acceptable for the information being conveyed. Yes No

Relevance The information within the source is relevant to my topic. Yes No

Authority The author has expert credentials or affiliations and/or the source is known, published, and reputable. Yes No

Accuracy The information is well researched and supported by evidence. Yes No

Purpose The source's slant or bias is not a hindrance to my project's focus. Yes No

The Prevalence of Checklists

Checklist approaches are widely used in academic libraries. A Google search for “CRAAP test” retrieves a significant number of academic library webpages and LibGuides. Many of them have been updated within the past year, which indicates current use. In contrast, a cursory review of two popular library instruction repositories, the ACRL Framework for Information Literacy Sandbox and Community of Online Research Assignments (CORA), revealed very few instances of checklist approaches; most of the recent lessons and assignments use approaches that emphasize critical practices such as contextualizing and verifying information. However, the fact that many academic library websites still focus on checklist approaches indicates that more change is needed.

5

Criticisms of the Checklist

Librarians have acknowledged problems with checklist approaches to teaching information evaluation for more than twenty years (for an early example, see Scholz-Crane, 1998). The next two slides show common criticisms. All of these problems can result in students falsely assessing the quality or truthfulness of a source (for evidence of this, see Scholz-Crane, 1998; Wineburg et al., 2020; Wineburg & McGrew, 2019). In today’s context, individuals or organizations can easily create websites that exist to spread disinformation but would pass the CRAAP test. Countless examples have been shared across social media during the COVID-19 pandemic. Someone who learned to evaluate information using a checklist would be unlikely to recognize disinformation without understanding the broader context.

6

Common Criticisms

The checklist approach is based on outdated criteria that often do not reflect quality.

• It uses criteria that librarians developed in the 1970s for print collections (Caulfield, 2018).
• It focuses on information such as domain (.org versus .com), layout, typos, ads, links/footnotes, contact information, and publication date (Breakstone et al., 2018; Caulfield, n.d.; Meola, 2004; Ostenson, 2014).
• It emphasizes details that can easily be manipulated, making evaluation misleading (Breakstone et al., 2018).

It takes too long to realistically teach or use.

• It can include dozens of questions and may give conflicting results (Breakstone et al., 2018; Caulfield, n.d., Meola, 2004).

7

Common Criticisms, continued

It asks questions that are often too complex for simple answers.

• It does not provide enough guidance to answer complex questions that often require expertise (Meola, 2004; Ostenson, 2014; Russo et al., 2019).

It encourages vertical reading rather than lateral reading.

• Students stay on a site to evaluate it, looking in places like the “About” page, which can be deceiving (Breakstone et al., 2018; Wineburg et al., 2020; Wineburg & McGrew, 2019).
• When using it, students “ignore or pay little attention to the broader context of an internet search” (Ostenson, 2014, p. 39).

Its simplistic structure is problematic.

• The yes/no structure encourages lower-order, dualistic thinking (Benjes-Small et al., 2013; Elmwood, 2020; Meola, 2004; Seeber, 2017).

8

Alternatives to the Checklist

A variety of alternative methods have been developed for teaching information evaluation. Some aim to transform checklists into something more critical, while others use a journalistic approach that encourages lateral reading (e.g., see Figure 2 on this slide). The next three slides show three different approaches, along with their benefits and limitations. These alternative methods all use open-ended questions rather than the dualistic approach of a checklist.

9

Figure 2. SIFT infographic by Mike Caulfield, licensed under CC-BY 4.0.

Approach 1: Make the Checklist Critical

This approach involves adaptations that make the checklist more critical, incorporating open-ended questions and giving students agency.

Examples

• RADAR uses similar criteria to CRAAP but frames them as open-ended questions (Mandalios, 2013).
• Benjes-Small et al.’s (2013) constructivist approach asks students to generate their own criteria.
• Campbell and Malone (2016) integrated CRAAP with ACRL’s (2015) Framework for Information Literacy for Higher Education.

Benefits

• Involves more critical thinking than a traditional checklist.

Limitations

• Many students are already familiar with one or more checklists and may rely on those ideas when generating or using alternative criteria.

10

Approach 2: Use Journalistic Methods

This approach uses journalistic or fact-checking methods that teach students to investigate sources through contextualization and corroboration.

Examples

• Caulfield’s (2019) SIFT method asks people to “Stop, Investigate the source, Find better coverage, [and] Trace claims, quotes and media to the original context” (see Figure 2 on slide 9).
• Radom and Gammons (2014) used the Five Ws: who, what, when, where, and why.
• Elmwood (2020) aimed to make the journalistic approach more accessible to students by mapping it to the ACRL Framework and Bloom’s taxonomy.

Benefits

• Encourages lateral reading.
• Often uses mnemonics that help students remember the steps or habits.

Limitations

• Tends to work best for mass media and may be less effective for some other types of information.

11

Approach 3: Focus on Certain Facets

This approach involves a wide variety of methods that focus on concepts such as authority, context, or emotions.

Examples

• Russo et al. (2019) focused on the authority and format of sources.
• Faix and Fyn (2020) provided ideas for using any of the six frames in the ACRL Framework to teach information evaluation.
• Lynch and Hunter (2019) recommended addressing the emotional aspects of evaluating information.

Benefits

• Allows for in-depth exploration of particular facets of evaluation, which can lead to deeper understanding.

Limitations

• Scaffolding may be required in order to teach more facets over time.

12

Barriers to Using Critical Approaches

In my review of the literature on teaching information evaluation, a number of barriers to using alternative methods emerged:

• Beyond the checklist, librarians may be unsure of what approaches to use (Caulfield, 2018).
• More critical approaches often require more time than a one-shot (Elmwood, 2020) and integration across curriculum (Breakstone et al., 2018).
• Critical approaches often involve polarizing topics, which can be challenging to address (Miller et al., 2019).
• Instructors may not be very familiar with the ways participatory information systems work (Fister, 2021) and therefore may not be confident in teaching about them.
• In cases where evaluating sources is not the primary focus of a lesson, librarians may desire a simple set of instructions they can provide as a handout.

​

13

Misinformation and Disinformation

In a time of significant political polarization, misinformation and disinformation present complex challenges. Conspiracy theories and distrust in science and experts are ubiquitous; this has been especially apparent during the COVID-19 pandemic. Fister (2021) argued that information literacy teaching needs to incorporate “the architectures, infrastructures, and fundamental belief systems that shape our information environment, including the fact that these systems are social, influenced by the biases and assumptions of the humans who create and use them” (The Search for Truth section). The ACRL Framework provides a way into these topics, and many alternative approaches to teaching information evaluation use the Framework’s threshold concepts.

14

Changing the Broader Approach

Fister (2021) noted “the lack of consistent instruction about information and media literacy across students’ educational experience” (What Went Wrong? section). A single lesson is not enough; the approach to information literacy needs to shift at all levels of education. This may seem daunting, especially given that most librarians have little influence over curriculum. Sharun and Smith (2020) provided an example of building stronger connections with faculty through educational development. They taught faculty workshops on fake news and digital literacy, which led to faculty incorporating some of these concepts into their own teaching. Examples such as this show ways that librarians can contribute to curricular change, if sometimes indirectly.

15

Reflection Questions

Do you use a checklist approach when teaching information evaluation? If so, why? If not, what approach(es) do you use?

If you do use a checklist, what support or resources would you need in order to shift to a different approach?

(How) can librarians realistically reduce the impacts of misinformation and disinformation?

What role can librarians play in curricular integration of information literacy and media literacy?

16

References 1 of 3

Association of College and Research Libraries. (2015, February 9). Framework for information literacy for higher education.

Benjes-Small, C., Archer, A., Tucker, K., Vassady, L., & Whicker, J. (2013). Teaching web evaluation: A cognitive development approach. Communications in Information Literacy, 7(1), 39-49.

Blakeslee, S. (2004). The CRAAP test. LOEX Quarterly, 31(3), 6-7.

Breakstone, J., McGrew, S., Smith, M., Ortega, T., & Wineburg, S. (2018). Why we need a new approach to teaching digital literacy. Phi Delta Kappan, 99(6), 27-32.

Campbell, P., & Malone, D. (2016). Frameworking CRAAP: How we’re correlating the ACRL Framework to content evaluation with the CRAAP test in our instruction practices. Georgia International Conference on Information Literacy.

Caulfield, M. (n.d.). The problem with checklist approaches. Check, Please! Starter Course. Retrieved March 4, 2021.

Caulfield, M. (2018, September 14). A short history of CRAAP. Hapgood.

Caulfield, M. (2019, June 19). SIFT (the four moves). Hapgood.

Elmwood, V. (2020). The journalistic approach: Evaluating web sources in an age of mass disinformation. Communications in Information Literacy, 14(2), 269-286.

17

References 2 of 3

Faix, A., & Fyn, A. (2020). Framing fake news: Misinformation and the ACRL Framework. Portal: Libraries and the Academy, 20(3), 495-508.

Fister, B. (2021, February 3). Lizard people in the library. Project Information Literacy.

Lynch, K., & Hunter, S. (2020). Conflicting authority: Using the Trump administration’s responses to the EPA climate assessment report to teach information literacy. Reference Services Review, 48(1), 201-216.

Mandalios, J. (2013). RADAR: An approach for helping students evaluate Internet sources. Journal of Information Science, 39(4), 470-478.

Meola, M. (2004). Chucking the checklist: A contextual approach to teaching undergraduates web-site evaluation. Portal: Libraries and the Academy, 4(3), 331-344.

Miller, S. D., Ording, G. J., Tans, E. D., & Vergara, C. E. (2019). “The earth is flat” and other thresholds: A critically reflective cross-disciplinary conversation in the post-truth era. In A. Baer, E. Stern Cahoy, & R. Schroeder (Eds.), Libraries promoting reflective dialogue in a time of political polarization (pp. 267-294). Association of College & Research Libraries.

Ostenson, J. (2014). Reconsidering the checklist in teaching Internet source evaluation. Portal: Libraries and the Academy, 14(1), 33-50.

18

References 3 of 3

Radom, R., & Gammons, R. W. (2014). Teaching information evaluation with the five Ws: An elementary method, an instructional scaffold, and the effect on student recall and application. Reference & User Services Quarterly, 53(4), 334-347.

Russo, A., Jankowski, A., Beene, S., & Townsend, L. (2019). Strategic source evaluation: Addressing the container conundrum. Reference Services Review, 47(3), 294-313.

Scholz‐Crane, A. (1998). Evaluating the future: A preliminary study of the process of how undergraduate students evaluate web sources. Reference Services Review, 26(3/4), 53-60.

Seeber, K. (2017, March 18). Wiretaps and CRAAP.

Wineburg, S., Breakstone, J., Ziv, N., & Smith, M. (2020). Educating for misunderstanding: How approaches to teaching digital literacy make students susceptible to scammers, rogues, bad actors, and hate mongers (Working paper A-21322). Stanford History Education Group, Stanford University.

Wineburg, S., & McGrew, S. (2019). Lateral reading and the nature of expertise: Reading less and learning more when evaluating digital information. Teachers College Record, 121(11), 1-40.

19

Icons are from Hand-Drawn Goods, licensed under CC-BY 2.5