Studying Learning with Digital Badges

Daniel Hickey

Research and evaluation are contentious topics in education.  This is because people disagree on what counts as “evidence” and what methods count as “scientific.” A 2001 report by the National Research Council argued that the “gold standard” of scientific educational research is randomized experimental trials. But the NRC also recognized that many of the most important ideas that might be tested in experimental research are unlikely to be discovered in experimental studies. This seems certain to be the case with digital badges in education.

The research designs introduced here are tentative and are intended to begin shaping conversation about where the systematic knowledge associated with digital badges is going to come from. The other three sets of badge design principles are more general characterizations of the specific practices that emerged across the thirty projects as they figured out how to use digital badges in their contexts. These principles should be quite useful for others who wish to use badges—particularly when the badges are linked to examples in projects and to relevant research resources. But these other badges design principles are not offered as “hypotheses” to be tested in experimental studies. Even if they did present testable hypotheses, the results would probably not generalize from the context where the experiment was conducted to other badging contexts where those findings might be applied. Rather these badges design principles are intended to be useful guidelines to help build systematic knowledge within the nascent badging community and to help newcomers to digital badges find and use the most relevant insights.

Research and Evaluation of Digital Badges

Thanks to the DML competition and extensive media coverage, many schools and programs are considering using digital badges. This means that many are also beginning to ask about the research evidence concerning the effectiveness of digital badges. Digital badges are so new that there are very few published studies because just a handful of studies have yet to make it through the peer review process. Grant and Shawgo’s (2013) annotated bibliography provides a nice summary of current badges research and provides additional relevant resources from other contexts. After the initial badges competition, HASTAC announced a separate research competition to study digital badges and made awards to five badges research projects. Some of these will be discussed below.

Few of the awardees included any formal research or evaluation studies in their original proposals. Notably, the DML 2012 competition did not require that proposals include detailed evaluation plans. This was probably a wise decision because requiring detailed evaluation plans may have led projects to prematurely search for “summative” evidence that badges “worked” before they had a chance to maximize the formative potential of digital badges to support learning. However, interviews with project leaders whose badge systems are now in place revealed that many were starting to think quite seriously about the sorts of studies they might conduct.

The issue that this paper addresses is that most of the project leaders were unclear as to where they might start in studying learning with digital badges  And only a few of them had even begun to grapple with the far-reaching idea of using the evidence contained in the digital badges in their research. Compare to the other three categories of principles uncovered in the DPD project, the principles proposed here are quite tentative, and draw on a broader set of ideas about educational research.

Three Important Distinctions for Studying Digital Badges

Attempting to makes sense of the possible kinds of studies that might be carried out with digital badges revealed three dimensions for thinking about research: systematicity, purpose, and evidence.

Systematicity. Arguably, the distinguishing feature of “research” is that it is systematic. Research involves systematically gathering some sort of evidence and attempting to document things in a way that could inform others. The design principles that the DPD project is identifying for recognizing, assessing, and motivating learning are mostly not coming out of systematic studies. In other words, the thirty projects are systematically developing badging practices, rather than more general principles. The DPD project is attempting to capture this more informal knowledge as it emerges as teams get their badge systems up and running.

Purpose. Building on the existing assessment literature, one can distinguish between summative studies “of badges” and formative studies “for badges.” Summative studies aim for a more naturalistic examination of the way the world is, while formative studies are designed as more interventionist efforts to change things. While most summative studies are intended to be formative, they do so less directly. One can also distinguish transformative research that examines how entire learning ecosystems are changed or created around badges.

Evidence. There is a distinction between studies that do not use the evidence of learning contained in digital badges and studies that do use this evidence. What makes digital badges unique is that they contain the actual evidence (or links to evidence such as artifacts produced by learners) to support particular claims of proficiency or accomplishment. There is usually a lot of negotiation involved in deciding what learning should be recognized with badges and how that learning will be assessed. As such, the evidence contained in badges will embody the values of the program or organization that issued them. As the DPD project is learning, a number of the projects ended without a functional badging system because projects simply could not manage to negotiation the claims, evidence, and assessments to associate with their badges. This seems to bolster the credibility of the information of the information in the other projects that were able to negotiate these issues. Taken together, these observations suggest that information contained in digital badges has enormous potential for summative, formative, and transformative research on learning.

Focusing on systematic studies and crossing purposes and evidence yields six research designs shown in Table 1.

Table 1: Six badge research designs.

Purpose

Using Conventional Evidence

Using Evidence in Badges

Summative

1. Research OF Badges

4. Research WITH Badges and

OF Badges

Formative

2. Research FOR Badges

5. Research WITH Badges and

FOR Badges

Transformative

3. Research FOR Ecosystems

6. Research WITH Badges and

FOR Ecosystems

 

Principles Within This Category

The following descriptions of each research design draw on selected examples from the DML competition as well as the studies being conducted by the awardees in the 2013 HASTAC Badges Research Competition.

Research OF Badges

Summative studies of digital badges are likely to be the largest category of badges research. Some will rely more on interpretive methods and qualitative evidence. For example, HASTAC Badges Research awardee Katie Davis (University of Washington) is studying how students and teachers in the Providence After School Alliance experience the badges used to give high school credit for expanded learning opportunities. Davis and her team will use interviews, questionnaires, and observations to explore (a) how badges fit in the academic and peer culture, (b) the role that badges play in motivation and achievement, and (c) whether badges connect in-school and after-school experience. Likewise, one of the studies being carried out by HASTAC Badges

Research awardee Jan Plass (New York University) falls in this category. Plass and colleagues will video record game play in publicly available games with and without digital badges. They will then analyze those recordings for trends and insights into participants’ perceptions and valuations of badges, and for changes in gameplay patterns due to badges. Other summative studies of badges might rely more on correlational methods and focus on individual differences and variables. In one of the first published peer-reviewed studies of digital badges, Abramovich, Schunn, and Higashi (2013) explored mastery-based and participation-based badges in an intelligent tutoring system for teaching proportional reasoning in mathematics. They measured self-reported motivation toward mathematics before and after the game, pre-achievement of proportional reasoning, and opinion toward badges. Correlational analyses revealed both positive and negative effects of badges on learner motivation, and that these finding interacted in turn with student ability and types of badges. The Badge Impact Survey (BIS) that Jan Plass will develop based on the results his initial observational study promises to be quite useful in this class of studies.

Other studies of the impact of digital badges will use experimental methods, such as creating different versions of the same types of badges issued. For example, the final study that Plass has proposed will modify a geometry game to examine the impact of two different types of badges. They will compare mastery badges (based on players’ own progress mastering learning goals) and performance badges (based on players’ performance relative to others). They will examine impact of the different badges on a range of individual outcomes, including motivation and learning. This study promises to provide generalizable principles about the impact of these two common types of badges in game-based learning environments. Other summative studies will be more consistent with typical program evaluations. While DML awardees were not required to include formal evaluations of their badging programs, many of them are now planning to evaluate the impact of badges as part of their larger organizational mission.

Research FOR Badges

Other studies will formatively intervene more directly in badge system design. One distinctly formative effort is the study proposed by HASTAC Badges Research Awardee Jim Diamond of the Educational Development Center. Diamond has already been working intensively with the DML/Gates 2012 Awardees Who Built America? (WBA) teacher mastery badge system. Diamond’s study is asking some of the same questions as Davis’ study of PASA. For example Diamond is asking about the role that WBA badges play in teacher professional development, and examining the ways that badge-related activities influence the development of an online teacher professional development community.

What pushes this research into the formative category is that Diamond is asking these questions while directly participating in efforts to build the badging system and the online professional development network. Studying things as they are changing quickly becomes complicated. And studying one’s own practice makes it hard to be “objective.” Diamond certainly recognized this in his proposal. This is why he is using design-based research (DBR) methods. As articulated by Paul Cobb and colleagues in 2003, DBR builds “local” theories in the context of iterative refinements of practice. Generally speaking, DBR studies start with some relatively general design principles for getting from the current state of affairs to the desired state of affairs. The back and forth process of translating the general principles into specific features yields specific design principles. Importantly, this process also reveals the key aspects of the learning context that support the specific design principles. It is this “embodiment” of the design principles in learning contexts that is presumed to generate useful insights that others can readily build on (Sandoval, 2004).

Two ongoing expansions of badging efforts should offer numerous opportunities for systematic formative research of digital badges. A number of researchers and graduate students were involved in efforts to design badge systems for the 2013 Summer of Learning in Chicago, and many will be involved in perhaps as many as a dozen similar citywide efforts in Summer 2014. Another context in which extensive formative research for badges is being carried out is a collective of informal learning organizations in New York City known as HiveNYC. While it is beyond the scope or timeframe of the DPD project to track of all of these efforts, it appears certain that new models of practice for formative studies of digital badge systems will emerge from them.

Research FOR Ecosystems

Many projects are using digital badges to create new learning ecosystems or transform existing ones. Some of the projects are beginning to study this process systematically. Consider the pilot study carried out by Global Kids of a new badging system for their youth programs. A DML award paired them with DML Badge System awardee Learning Times to implement BadgeStack in Global Kids’ Race to the White House and Virtual Video Project programs. The report of the pilot study provides some examples of what this might look like. The report of the pilot study describes how badges impacted the educational programs that Global Kids had already developed. For example, they found that:    

Global Kids youth leaders received confirmation 48 times that evidence submitted of their work met the requirement of one of thirteen different educational objectives in their programs. At the same time, youth leaders received confirmation ten times that their evidence did not meet the requirements. Both took extra time—for the youth to submit the evidence and the GK staff to review and evaluate—but the goal of providing formative assessment was significantly advanced (2012, p. 6).

The report explained that this sort of assessment had never been carried out in the educational programs that Global Kids offer. Other systematic studies of the transformational effects of badges on ecosystems are likely to emerge in the Summer of Learning and various Hive projects. Another example is the dissertation study being conducted by Rafi Santo. A grant from the New York Community Trust is supporting his extended study of the diffusion of innovations in the Hive NYC. This study is not focusing specifically on digital badges. But a DML award to Global Kids is ensuring that badges are systematically implemented across the Hive NYC community. This and other such efforts promise to provide more specific research design principles for studying the creation and transformation of learning ecosystems via badges and other specific innovations.

Formative studies of entire learning ecosystems are incredibly complex. There are many variables to consider, numerous principles and features to be refined, and many methods that might be used. There are also complex issues that arise when attempting to link the learning of students/mentees with the learning of teachers/mentors. While Jim Diamond’s study certainly has some of these characteristics, it seems like he made a wise decision to tame some of that complexity by staying within the DBR framework. However, as the badges community matures, it is certainly going to need to tackle this complexity. Fortunately, a new strand of DBR known as Design-Based Implementation Research (DBIR, Penuel et al., 2011) aims to address these additional challenges. In particular, DBIR explicitly addresses (a) the existence of multiple stakeholders with different perspectives, (b) the crucial and unique role of educators and mentors in DBR, and (c) a concern with developing capacity for sustaining change in systems.

Research WITH Badges and OF Badges

Using the evidence contained in badges offers new opportunities for summative research of badges. This includes studies of the credibility of claims made in badges. This question naturally has come up a lot around digital badges. Jacobs, in a 2012 article in US News & World Report suggested badges might someday overturn the monopoly that colleges currently hold on formal credentials—but only if badges are proven credible. As badges begin to function as more formal credentials, employers and college admissions officers are wondering about the reliability of the assessments behind the badges and validity of the claims made in badges. Some have noted that the credibility of conventional credentials (grades and transcripts) is seldom systematically scrutinized. Nonetheless, more formal badges are likely to trigger studies using conventional criteria from educational and psychological testing (e.g., internal reliability, construct validity, generalizability, etc.). Mozilla’s Carla Casilli (2012) argues that being web-enabled means that the validity of the claims made in any badges will ultimately be crowdsourced. This means that evidence from formal reliability and validity studies might be meaningless if relevant personal or professional networks collectively ignore or dismiss that evidence. Casilli points out that if this turns out to be true efforts to understand the credibility of badges will have to look beyond the validity literature to consider research about the credibility of information on the Internet. One promising example is Fogg’s (2003) taxonomy of credibility, which includes presumed, surface, reputed, and earned credibility.

The evidence contained in digital badges has many other potential uses. The aforementioned pilot study of badges at Global Kids provides initial examples of the how programs can use the evidence to study how learning occurs in their programs. Before Global Kids introduced badges, their primary evidence of learning in program evaluations were summaries of blog entries that students were asked (but not required) to make. With digital badges it was simple to link to a detailed description of the badges that were offered to program participants. Additionally, the details of who earned what badges provide a surprisingly comprehensive picture of the learning that was supported by the program. Examining the order in which badges were earned also allowed Global Kids to begin studying the paths that learners took through their programs. Given the challenges that many schools and programs face in evaluating and studying learning, the introduction of digital badges seems poised to unlock enormous potential in this regard.

Research WITH Badges and FOR Badges

The evidence contained in digital badges also has the potential for systemic efforts to formatively improve badge systems. Consider for example, the work of Stacy Kruse, Creative Director of DML 2012 awardee Pragmatic Solutions. Kruse is collaborating with the Digital On-Ramps project in Philadelphia and several educational initiatives at the Corporation for Public Broadcasting. As Kruse put it in response to an interview questions about badges research, “before I started working with digital badges, I was working on learning analytics.” This kind of experience has left Kruse and colleagues quite enthusiastic about building learning analytics directly into the badging systems they are building, and using those results to dynamically refine what badges are available, how they are displayed, etc.

Interviews with other DML awardees uncovered some other promising efforts to use the evidence in badges to transform badging systems. GoGoLabs CEO Lisa Dawley and the Planet Stewards project are using badges to connect educational content from the National Oceanic and Atmospheric Administration to the Next Generation Science Standards. One of their challenges is mapping the game-like curricular “quests” to the standards. Such mapping is notoriously difficult and a major obstacle to standards-based reform. Curricular activities naturally touch on multiple standards, and systems need redundancy so that students and teachers can select from multiple activities. Because badges can be more specific and because they contain actual evidence of learning, they open up entirely new formative possibilities for mapping. This same evidence can then be used summatively to examine the learning trajectories that students take.

Research WITH Badges and FOR Ecosystems

Eventually researchers are likely to begin using the evidence in digital badges to systematically study and improve entire learning ecosystems. In this way it seems possible that digital badges might ultimately transform the entire learning analytics movement. But this seems unlikely to even get started until clear research design principles for summative and formative studies using the evidence in badges emerges.

Table 2. General Design Principles for Studying Learning in Digital Badge Systems

General Design Principle

Description

Research OF badges

Often using interviews and surveys, this research aims to study the badge system's impact and integration into learners' lives.

Research FOR badges

Research is explicitly intended to feed back into the badge system to iterate its design and better achive its goals.

Research FOR ecosystems

Transformative research examines how entire learning ecosystems are changed or created around badges.

Research WITH badges & OF badges

Research uses the evidence in badges to study the system, including the validity of its assessments and how learners moved through it.

Research WITH badges & FOR badges

Implement systematic research to improve badge systems, by analyzing the badges that were issued.

Research WITH badges & FOR ecosystems

Eventually studies may analyze badges across wide badge ecosystems to find ways of improving the function of entire learning networks and culture.

References

Abramovich, S., Schunn, C., & Higashi, R. M. (2013). Are badges useful in education?: it depends upon the type of badge and expertise of learner.Educational Technology Research and Development, 1-16.

Casilli, C. Badge System Design: what we talk about when we talk about validity.

 (2012, May 21). Retrieved from http://carlacasilli.wordpress.com/2012/05/21/badge-system-design-what-we-talk-about-when-we-talk-about-validity/

Cobb, P., Confrey, J., diSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9–13.

Fogg, B. J. (2003). Persuasive technology: Using computers to change what we think and do. Amsterdam: Morgan Kaufmann Publishers.

Global Kids. (2012). Badges for learning: An abridged recent history. Retrieved from http://www.olpglobalkids.org/content/badges-learning-abridged-recent-history

Grant, S. & Shawgo, K.E. (2013). Digital Badges: An Annotated Research Bibliography.  Retrieved from http://hastac.org/digital-badges-bibliography

Penuel, W. R., Fishman, B. J., Cheng, B. H., & Sabelli, N. (2011). Organizing research and development at the intersection of learning, implementation, and design. Educational Researcher, 40(7), 331-337.

Sandoval, W. A., & Bell, P. (2004). Design-based research methods for studying learning in context: Introduction. Educational Psychologist, 39(4), 199-201.

Shavelson, R. J., & Towne, L. (Eds.). (2002). Scientific research in education. National Academies Press.