Whose Values Count?
Assessment as a Mirror of What We Prize in Media Literacy
Renee Hobbs
EMAIL: hobbs@uri.edu
Media Education Lab · University of Rhode Island USA
REDMIL Doctoral Summer School, Brussels
1 September 2026
“
Digital and media literacy competencies refuse to stand still long enough to study them.
Hobbs, R. (2023). Issues of pedagogy, alignment, and context in assessing measures of media literacy. In N. Landry & P. Fastrez (Eds). Media Literacy and Media Education Research Methods: A Handbook (pp. 200 – 219). Routledge.
Media Education Lab · ReDMIL 2026
--Hobbs, 2023
SMOKING MEDIA LITERACY (SML) SCALE
Primack, Gold, Switzer, Hobbs, Land & Fine (2006) · 22 items
“When people develop movies or TV programs, individual camera shots are carefully designed.”
Strongly disagree
Disagree
Neither
Agree
Strongly agree
Media Education Lab · ReDMIL 2026
Media Literacy in Public Health
Validated and reliable self report Likert-scale items
SMOKING MEDIA LITERACY (SML) SCALE: What the scale measured
22
items on the scale
Self reported practice of critical analysis with items exploring:
Authors and Audiences, Messages and Meanings, Representations and Realities.
SMOKING MEDIA LITERACY (SML) SCALE: What the scale measured
22
items on the scale
34%
lower susceptibility to smoking, per point gained
Scale documents self-reported practice of critical analysis with items exploring:
Authors and Audiences, Messages and Meanings, Representations and Realities.
SMOKING MEDIA LITERACY (SML) SCALE: What the scale measured
22
items on the scale
34%
lower susceptibility to smoking, per point gained
0
items about creating media,
collaborating, or taking action
Assessing media literacy competencies, skills, knowledge and programs is not a neutral technical exercise.
It is a reflection of sometimes competing values.
What was decided about your project
before you arrived — and by whom?
PAIR SHARE: 90 seconds each
Your supervisor’s existing grant
The national research council’s priority area
The framework everybody in your subfield cites
The population your lab already has access to
Media Education Lab · ReDMIL 2026
Something else
What was omitted?
Researchers have blind spots about the different forces that shape knowledge production.
Media Education Lab · ReDMIL 2026
Research Decisions Have Values Consequences
Examine two recent research projects
1,275 teachers surveyed in 2025
17 educators · 138 youth (2026)
Media Education Lab · ReDMIL 2026
Considering How Researcher Choices Get Made
| Massachusetts - 2025 | West Bank - 2026 |
Who decided on the goal? | | |
How were measures and methods selected? | | |
How were findings framed? | | |
Media Education Lab · ReDMIL 2026
What was omitted?
MASSACHUSETTS DEPARTMENT OF ELEMENTARY & SECONDARY EDUCATION
FUNDER GOAL: Media literacy as civic formation.
Media Education Lab · ReDMIL 2026
RESEARCHER GOAL: Large-scale implementation of new measures of ML prevalence & use of new methods for synthesizing expert knowledge
MASSACHUSETTS
How were measures
and methods selected?
MASSACHUSETTS
What is media literacy?
INTERVIEW FINDINGS: What 46 Massachusetts educators thought “media literacy” meant
Teaching with media
Online safety
Critical analysis of news
Technical skills
Interview findings confirm that teacher understanding of media literacy is uneven and varies widely.
Media Education Lab · ReDMIL 2026
Interpreting Evidence
MASSACHUSETTS
Fact checking
Asking critical questions
Knowledge about media industries
Creating media
Reflecting on media’s role in society
MASSACHUSETTS
Deciding on a sampling strategy
Representative sample or convenience sample?
Sampling strategy
We chose not to use a representative sample of Massachusetts teachers.
MASSACHUSETTS
Instead, we used a large convenience sample which relied on volunteers who chose to participate.
How were measures
and methods selected?
FEAR: Representative sample of K-12 teachers may show there is low awareness and low interest in media literacy.
HOPE: Convenience sample may show scope and capacity of K-12 teachers as stakeholders for media literacy statewide.
MASSACHUSETTS
Selecting methods,
instruments and approach
Producing findings that showed student ML incompetence created a sense of urgency that ultimately did not increase the quality of media literacy education in US public schools
Producing findings that showed student ML incompetence created a sense of urgency that ultimately did not increase the quality of media literacy education in US public schools
Deficit findings generated “quick fixes” that were not sustainable
Urgency led to unfunded mandates over building capacity
Alarm alienates the people who must do the work
Crisis attention has a short half-life.
.
“
Digital and media literacy competencies refuse to stand still long enough to study them.
Hobbs, R. (2023). Issues of pedagogy, alignment, and context in assessing measures of media literacy. In N. Landry & P. Fastrez (Eds). Media Literacy and Media Education Research Methods: A Handbook (pp. 200 – 219). Routledge.
Media Education Lab · ReDMIL 2026
--Hobbs, 2023
21
Measure educator perceptions of implementation prevalence
Media Education Lab · ReDMIL 2026
Interpreting Evidence
MASSACHUSETTS
Survey participants were asked to estimate how many of the learners in their school get exposure to 21 instructional practices, and they made their estimates on a 10-point scale, using a slider bar that ranged from 1 = “hardly any” to 10 = “nearly all”.
LEAST LIKELY INSTRUCTIONAL PRACTICES
7.99
7.68
7.67
4.26
4.25
4.24
MOST LIKELY INSTRUCTIONAL PRACTICES
Some instructional practices
of media literacy are widely
used in Massachusetts
Increased attention to
teacher education can increase the range & diversity of instructional practices
FOR POLICYMAKERS
Delphi method: a structured communication method that relies on a panel of experts, often used to reach consensus and develop professional guidelines
MASSACHUSETTS
How were measures
and methods selected?
ROUND 0
Panel writes its own definitions
ROUND 1
5 reviewed, 16 added,
145 comments
ROUND 2
Rate 10 definitions:
best / worst / neither
ROUND 3
See everyone’s votes,
revise or hold
What the design assumes: Dialogue and discussion among experts can lead to consensus
Media Education Lab · ReDMIL 2026
Delphi Study
MASSACHUSETTS
Ask the Experts: What is media literacy?
“What is the best definition of media literacy in 2025?”
97 media literacy experts and experienced education professionals, Round 3
Media Education Lab · ReDMIL 2026
54%
2%
“What is the best definition of media literacy in 2025?”
97 media literacy experts and experienced education professionals, Round 3
“…a process of engaging with media through critically examining representations, systems, structures, ideologies, and power dynamics that shape and reproduce culture and society. It is an inquiry-based process for analyzing and creating media by interrogating the relationships between power and knowledge.”
19%
26%
RATED BEST
RATED WORST
“What is the best definition of media literacy in 2025?”
97 media literacy experts and experienced education professionals, Round 3
“…a process of engaging with media through critically examining representations, systems, structures, ideologies, and power dynamics that shape and reproduce culture and society. It is an inquiry-based process for analyzing and creating media by interrogating the relationships between power and knowledge.”
19%
26%
“This is a pretty good definition but a bit too jargony for public uptake.”
“I don't like the inclusion of power in this definition -- especially for younger students. This pre-disposes it toward foregrounding of power struggles in society, which need context that is beyond the MLE class.”
“This one is sure to raise partisan alarm and attract lots of opposition. It's scary sounding.”
Consensus & dissensus
are part of the ML landscape
Disagreement about definitions and pedagogy are a longstanding feature of the field. Perceptions about definitions and implementation practices reflect the differential positionality of those who participate in the media education field.
FOR POLICYMAKERS
Media Education Lab · ReDMIL 2026
Interpreting Evidence
MASSACHUSETTS
ML experts had some consensus regarding prioritizing practical advice. Over two rounds, panel members were asked, “Which of these items of advice should be recommended to K-12 educators regarding teaching media literacy?” They placed items in rank order to represent top recommendations.
The use of rank order makes it seem like there is more controversy about pedagogical priorities
MOST FEASIBLE
Guidance and support
from state agency
MODERATELY FEASIBLE
Funding for PD and
Pilot programs
MORE CHALLENGING
New ML assessments and Teacher credentials
What’s missing? RENEE’S THOUGHT EXPERIMENT
Sort the identical data by “which students in which communities have the least access,” and a completely different report could be written, using the same evidence with a different value frame.
Systemic change is relegated to the status of “more challenging”
Media Education Lab · ReDMIL 2026
Interpreting Evidence for Policymakers
MASSACHUSETTS
Considering How Researcher Choices Get Made
| Massachusetts | West Bank |
Who decided on the goal? | ADVANCE STATE EDUCATION POLICY Demonstrate how media literacy can be implemented in education across subject areas and grade levels | |
How were measures and methods selected? | STAKEHOLDERS & EXPERTS Delphi method produces expert consensus & dissensus and survey/interview data document uneven implementation of ML instructional practices | |
How were findings framed? | FEASIBILITY ANALYSIS Data analysis produces direction for policymakers with proposed structural changes presented as most challenging | |
Media Education Lab · ReDMIL 2026
What was omitted?
US EMBASSY IN JERUSALEM (FORMER OFFICE OF PALESTINIAN AFFAIRS)
Media Education Lab · ReDMIL 2026
FUNDER GOAL: Media literacy as combatting disinformation
RESEARCHER GOAL: Document evidence of cultural adaptation of ML pedagogy in wartime
WEST BANK
How to measure change in participant knowledge, attitudes,
beliefs and behavior over time?
17 higher ed faculty
138 youth participants
How were measures
and methods selected?
Four-wave faculty survey
Over 12 months
8 interviews and focus groups
faculty and students asked about program quality
300+ pages of Arabic transcript
all nine 3-hour youth workshops
Youth post-only survey
Padlet post counts
Engagement with creating media
Youth-created media
thematic analysis of the digital content they created
Media Education Lab · ReDMIL 2026
WEST BANK
short questionnaire focused on perceptions of relevance
Needs assessment
Interviews about the context of media use & education background
How were measures
and methods selected?
WEST BANK
How to measure change in faculty knowledge, attitudes,
beliefs and behavior over time?
We emphasized the measurement of confidence in the use of ML instructional practices,
not content knowledge about combatting disinformation
How were measures
and methods selected?
WEST BANK
Nine faculty teams developed & implemented media literacy workshops for young adults
Using ”The Power of Two,” teams of Palestinian faculty designed and implemented a 3-hour online workshop for Palestinian young adults, aged 16 – 25
Researchers provided a foundational framework in the training program and then faculty participants built and delivered a relevant educational experience.
Media Education Lab · ReDMIL 2026
FINDINGS
Cultural adaptation varied according to the disciplinary backgrounds of the faculty.
The 138 student participants did not have the same learning experience, making it difficult to measure program impact
WEST BANK
How to measure change in faculty knowledge, attitudes,
beliefs and behavior over time?
Evidence of intellectual humility
after educators implemented their programs
Four waves of data collection over 12 months
How were measures
and methods selected?
WEST BANK
How to measure change in faculty knowledge, attitudes,
beliefs and behavior over time?
Lower self-report scores
after educators designed & implemented their youth ML programs
How were measures
and methods selected?
WEST BANK
Students developed social media posts to
synthesize their learning, and we coded thematically
Media Education Lab · ReDMIL 2026
A paid social media campaign enabled “youth voice” to gain visibility in the West Bank, fulfilling the funder’s goal of public diplomacy
WEST BANK
Students learned what we did not teach the teachers
Media Education Lab · ReDMIL 2026
Why Media Literacy as ARMOR & SHIELD?
76,000+ deaths of Palestinians
270 media professionals
130 academics
560 humanitarian aid workers
59% of victims were women, children, and elderly
Two numbers from the same report
6,155,638
total reach of a recruitment campaign and a youth-created social media campaign across the West Bank, Gaza and Jerusalem
17
educators who completed the entire program: 11 webinars, 4 surveys, an implementation plan, a co-delivered workshop
The number the funder needs.
Interpreting Evidence
The number the field needs.
Blind Spots
“Internet instability”
“Electricity interruptions”
“Weak online connectivity”
FROM THE INTERVIEWS: Six of eight educators named one of these types of technical failure as the low point of the program experience.
In most research, a stable internet connection is a background assumption, the thing you don’t measure because it is always there.
Media Education Lab · ReDMIL 2026
“Media literacy education generally presumes a liberal-democratic context of relatively free expression and trustworthy institutions.”
Media Literacy in Palestine, Final Report, July 2026
What the funder asked for:
“Building Media Literacy and Combatting Disinformation”
What the scholarly literature told us:
A “media literacy of the oppressed” is critical analysis that is inseparable from analysis of power and dispossession
Media Education Lab · ReDMIL 2026
Blind Spots
Media Education Lab · ReDMIL 2026
We modeled strategies for teaching about media regulation by examining Israeli military law and its (many) limits on Palestinian freedom of expression
No educator in the program chose to focus on media regulation or issues of Palestinian freedom of expression in wartime in the workshops they developed for youth
WHAT WE EXPERIENCED
The scope and limits of teacher agency in wartime
Considering How Researcher Choices Get Made
| Massachusetts | West Bank |
Who decided on the goal? | ADVANCE STATE EDUCATION POLICY Demonstrate how media literacy can be implemented in education across subject areas and grade levels | PUBLIC DIPLOMACY Model instructional strategies for faculty and students to combat disinformation and understand propaganda shared by social media influencers |
How were measures and methods selected? | EXPERTS AND STAKEHOLDERS Delphi method produces expert consensus and survey and interview data document implementation of ML instructional practices | TEACHERS AS LEARNERS Surveys examine change in faculty knowledge and confidence over time; content analysis of workshop transcripts and youth-produced media demonstrate how ML concepts are inflected through cultural values. |
How were findings framed? | FEASIBILITY ANALYSIS Data analysis produces direction for policymakers with recommended activities to advance implementation and practice | VALUING CULTURAL ADAPTATION Train-the-trainer programs do not have to require fidelity to be effective. Teacher confidence in teaching ML gets recalibrated after direct ML teaching experience. Youth sense-making through media production reveals evidence of learning. |
Media Education Lab · ReDMIL 2026
What was omitted?
Considering How Researcher Choices Get Made
| Massachusetts | West Bank |
Who decided on the goal? | ADVANCE STATE EDUCATION POLICY Demonstrate how media literacy can be implemented in education across subject areas and grade levels | PUBLIC DIPLOMACY Model instructional strategies for faculty and students to combat disinformation and understand propaganda shared by social media influencers |
How were measures and methods selected? | EXPERTS AND STAKEHOLDERS Delphi method produces expert consensus and survey and interview data document implementation of ML instructional practices | TEACHERS AS LEARNERS Surveys examine change in faculty knowledge and confidence over time; content analysis of workshop transcripts and youth-produced media demonstrate how ML concepts are inflected through cultural values. |
How were findings framed? | FEASIBILITY ANALYSIS Data analysis produces direction for policymakers with recommended activities to advance implementation and practice | VALUING CULTURAL ADAPTATION Train-the-trainer programs do not have to require fidelity to be effective. Teacher confidence in teaching ML gets recalibrated after direct ML teaching experience. Youth sense-making through media production reveals evidence of learning. |
Media Education Lab · ReDMIL 2026
What was omitted?
I manipulated project goals to focus research questions on issues of teacher agency
This leads to the subordination of scientific rigor that limits the generalizable knowledge claims that can be made from evidence
I frame findings in ways that celebrate the consensus and dissensus of a global and fast-changing field
What was omitted?
Researchers have blind spots about the different forces that shape knowledge production.
Because we work within institutional systems that shape our choices,
personal reflection about positionality must be a key feature of the research process.
Media Education Lab · ReDMIL 2026
Research Decisions Have Values Consequences
Media Education Lab · ReDMIL 2026
Why Do We Do Research?
Because of What We Don’t Know
Research is a learning process
that deepens intellectual humility
Wife
Mother
Grandmother
Teacher
Teacher Educator
Researcher
Advocate
Mentor
Field Builder
Media Maker
Global Citizen
Learner
Whose Values Count?
Assessment as a Mirror of What We Prize in Media Literacy
Renee Hobbs
EMAIL: hobbs@uri.edu
Media Education Lab · University of Rhode Island USA
REDMIL Doctoral Summer School, Brussels
1 September 2026