Introduction to the Badge Design Principles Documentation Project

Daniel Hickey and Nate Otto

Digital badges are visual symbols of credentials. They make specific claims about learning and can offer detailed evidence in support of those claims. The common structure of this information was formalized in the Open Badge Infrastructure (OBI). The Mozilla Foundation began initial planning of this structure in 2010; the OBI as it currently exists was formalized in 2012. The OBI defined how various types of metadata are embedded in badges and verified by issuers to tie each specific badge image to details on how it was earned. This allows any person or organization to issue, earn, display, and validate compatible digital badges. They are designed to be easily shared wherever learners want to showcase their accomplishments online, including websites, digital resumes, and social media.

Any organization that wishes to use Open Badges to recognize learning in their own context must build a badge system. At a minimum, such a system defines the various skills and achievements to be recognized and builds criteria to match, sets down the assessment methods to be used to decide when that criteria is met. In this set of papers, we argue that a fully considered badge system design will include these recognition and assessment decisions, but also will examine the motivational effects of the design and plan to collect and analyze data relevant to its context. Many decisions in these areas define a badge system for the long term. There are complex relationships between different design elements that should be unpacked and analyzed in advance, but they may also require modifications to the system as the project moves from design into implementation or after running the system in the real world.

As with any open technology, the possibilities for using digital badges are vast, and the range of considerations for effective use are not immediately clear. Digital badges open the possibility to recognize a wider range of learning than is feasible with traditional credentials like diplomas and degrees. This includes badges as micro-credentials for granular achievements as well as badges that cross boundaries to recognize learning that happens inside and outside of traditionally credentialed spaces.

Seeing a need to build a research community around using this technology for learning, Grant and Shawgo (2013) curated an extensive annotated bibliography of the research literature that appears relevant to digital badges. In her post “Badges Now,” Davidson (2013) provided an overview of the function and potential impact of badges, exploring the value provided by badges as alternative forms of assessment and credentialing. When executed well, badges have the potential to recognize diverse forms of learning and powerfully tell the narrative of students’ learning journey. Knight (2010) described a rationale for badges in her post ‘Certification’ Revisited,” raising open questions on the design of badges and how badges might fit into the educational landscape. Learning can happen anytime, anywhere. By providing evidence of learning, badges can show students’ trajectory as well as their ability. There is potential to provide more detailed information to others and ease the translation of credentials between communities.

To further explore how open badges can be effectively used in learning programs, the MacArthur Foundation, in partnership with Mozilla, HASTAC, and the Bill & Melinda Gates Foundation, sponsored the Digital Media & Learning (DML) competition that funded 30 projects to implement badges in pursuit of recognizing lifelong learning. These projects occupied niches across the spectrum from formal to informal learning environments and served programs in classrooms, museums, libraries, after-school programs, online learning platforms, and more.

The Design Principles Documentation Project

The DML badges competition organizers hoped to capture the experiences of the participating projects and draw conclusions that would be useful for future badge system design and research. This paper, from the Design Principles Documentation Project (DPD Project) came out of this effort. The DPD Project is documenting the practices developed for the DML projects’ badge systems, studying these practices, and distilling them into sets of general and reusable design principles in four categories: (a) recognizing learning, (b) assessing learning, (c) motivating learning, and (d) studying learning.

The project’s overall goal is identifying appropriate practices for using badges in particular learning contexts. This is important because the features that define contexts are what determine whether a particular practice is appropriate. It is complicated for those working with badges, because the practices and contexts are mostly new, they are evolving, and the practices and contexts within a project shape one another. The DPD Project authors believe, and hope to show through this investigation, that badge system design is not a matter of applying “best practices,” but is instead a process that involves tailoring practices to fit project goals and contexts. The DPD Project’s specific goals are to tame this complexity by describing general design principles for badge systems and showing evidence of how they work in particular contexts.

A badge system is unlikely to find the perfectly tailored practices that serve project goals on its first iteration. As Mozilla’s Carla Casilli explained:

Regardless of where you start, it’s more than likely you’ll end up somewhere other than your intended destination. That’s okay. Systems are living things, and your badge system needs to be flexible. You must embrace a bit of chaos in its design. (2012)

While acknowledging some possible chaos, prospective system designers may reach for resources that help give them a roadmap of how the different elements of badge systems work together. For these system designers, the DPD Project hopes to provide resources that will both help get their systems closer to the mark initially and provide examples of how other programs modified their practices to address contextual challenges that they may encounter. In addition, the project aims to provide a research framework for badge system managers to apply to studying their own systems and for future badge researchers to apply as they study systems externally.

Methods Used to Capture Badge Design Principles

The main findings are presented as four sets of design principles for use in digital badge systems, covering the categories of recognizing, assessing, motivating, and studying learning. Each principle is a distillation of related specific practices found among the DML projects. As they represent general versions taken out of their original contexts, it is necessary to recontextualize design principles to form them to the needs of new projects. The DPD Project’s description of principles and contextual practices interact is heavily informed by Design Based Research (DBR), as configured by Cobb, Confrey, diSessa, Lehrer and Schaulble (2003). In the terminology used in this report, general design principles are translated into specific practices through the construction of “local theories” that explain how design features of the intended practices serve project goals and function in context. Often projects’ local theories fit into subdivisions of the general principles, named specific principles.

Project Phases

The DPD Project is tracking how specific practices evolve in their contexts, from the initial design phase, through implementation, to becoming formal continuing practices that appropriately serve project goals.

Our project’s process started by identifying and naming intended specific practices for each of the DML competition projects’ badge systems through analyzing written proposals and conducting interviews. After collecting all the practices, team members clustered similar practices together and similar groups of practices together. These formed a two tier hierarchy of design principles with one or more groups of specific principles within each general design principle. The important feature of this structure is that projects’ specific practices are embedded in the local theories that define the function of that particular badge system, while general and specific principles could be represented by practices in a variety of badge systems.

With this structure in place, work began identifying how the projects’ practices changed as they moved from intended practices to the implementation stage, where their ideas became enacted practices. A final round of interviews followed, and as publishing begins on case studies of each of the DML projects, the final stage is to consider how the enacted practices solidified as continuing formal practices and how they relate to the identified design principles.

The lists of principles established in this project are by no means a complete set of the ways projects will use badges for learning, nor are the four categories identified necessarily a complete set of the functions of practices within badge systems as the overall ecosystem and culture around badges develops. There will likely be many interactions between practices that the findings from the DML competition do not anticipate. The authors of the DPD Project hope that this research can serve as a platform for future exploration of badge practices and that it can be adapted to meet the needs of future designers and researchers.

General Findings of the Design Principles Documentation Project

        The DPD Project is beginning to publish research findings, first as this preliminary draft report and its accompanying case study appendices, to be followed by refinements and a more complete set of project profiles of the DML competition badge systems. At this stage, we are hesitant to make many generalizations across all of these project, but there are a few findings that stand out.

 

Tensions Between Practices

One important general finding from the DPD Project concerns the relationships between the four different types of practices. It became clear looking across the projects that the broader ecosystem constrains the learning that badges can be used to recognize. Recognition practices, in turn, constrain assessment practices; recognition & assessment practices together serve to impact motivation. Finally the entire badging ecosystem, including the recognition, assessment, and motivation practices, impact the research designs that might be used to study digital badges. These tensions will be explored in each of the four following category chapters.

Tensions Between Different Approaches Within Practices

The second general finding from the DPD Project concerns different approaches to the same practices. The local theories developed in each project’s context can rely on widely varying foundations. Within each of the four categories identified by the project, there are very different (and potentially competing) assumptions about knowing and learning that might be used to inform those practices. Generally speaking, these different approaches reflect the three “grand theories” of knowing and learning.

Associationist perspectives. The most traditional grand theory is often labeled associationist. It has its roots in behaviorism but is well represented in the work of some cognitive scientists and many instructional systems technologists. This perspective assumes that knowledge consists of numerous small associations, which means the learning involves building and strengthening those associations. These perspective are best exemplified on “competency-based” badging systems like BuzzMath the focus on more procedural and factual knowledge that is assessed continuously using multiple-choice formats.

Rationalist perspectives. The second grand theory is rooted in modern “constructivist” perspectives that emerged in the 1980s and that a widely embraced among many cognitive scientists and educational and educational psychologists. This perspective assumes that knowledge consists of broader conceptual schema that are constructed when attempting to make sense of new information in the world. These perspectives are best exemplified by “inquiry-oriented” and “project-based” badge system. Who Built America is a good example of these kinds of badging systems, because both the teachers who earn badges in the project generally do so for completing projects that are then assessed by peers using a structured rubric.

Sociocultural perspectives. The third grand theory is rooted in contemporary sociocultural perspectives that emerged in the 1990s. These perspectives are less well known, but are strongly embraced by some cognitive scientists and many learning scientists. These perspectives assume that knowledge is fundamentally represented in social and cultural practices of groups of people, and therefore view learning in terms of increasingly successful participation in those social practices. This perspective is best illustrated by badge systems such as Supporter to Reporter Medals and Mouse Wins! that focus much of their effort on building a digital social network around their learning ecosystem. Within sociocultural perspectives, a strand of theories known as “situative” perspective argue that both associationist practices and rationalist practices can be understood as “special cases” of sociocultural practices.

Each of these three perspectives on knowing and learning has direct implications for supporting, recognizing, assessing, motivating, and studying learning. Particularly in the case of the first two perspective, some of these practices might be quite incompatible with each other. Proponents of the third perspective argue that it offers a way of resolving these tensions. Yet this raises complex issues and is not widely embraced by researchers and is difficult to comprehend for practitioners.

DPD Project Resources

In the current report, we offer four chapters, authored by the DPD Project team members who lead the investigation of each category. These chapters will introduce the design principles with selected examples from the DML projects. They will also present some of the relevant research to the function of digital badges in each category and discuss common tensions that affect how the principles can be translated into specific practices. Each chapter contains a table for quick reference of the design principles and lists of the projects that enacted them for further investigation.

We attach several case studies as appendices to this report. Each of these project profiles takes a deep dive into the intended, enacted and continuing practices of a badge system, their relationship to general design principles and a discussion of particularly illuminating challenges faced by the system.

We also include the first draft of an open educational resource (OER) produced by our team to assist with system design or analysis: a printable card deck of the design principles, suitable for remixing into new configurations on the tabletop.

How to Use These Resources in Research and System Design

For those who hope to implement digital badges for learning, the Design Principles Documentation Project describes the goal as identifying appropriate practices that serve project goals and function well in the project context. In addition, it involves determining how practices across different functions of the badge system interact.

To use these resources to design new badge systems, designers should first become familiar with the design principles across all four areas identified by this project and then try out thinking how they could be contextualized to fit the needs of the project at hand. See how various principles feel together, perhaps by rearranging principle cards on a table.

Start with the learning or achievement the system should recognize. Consider the type of evidence necessary to decide when to issue badges that make these learning claims. Move through the categories, and change your perspective to see how your developing system functions. Think about how it serves to motivate learning after deciding on a set of compatible recognizing and assessing principles, for example. Think about the sort of data that would be generated by the operation of the system and what data designers would need to improve it. When the last category is reached, circle back around or jump back and forth between categories and see if additions have introduced complexities that force reconsideration or would strain available resources. Make sure to write down any ideas that seem not to fit within one of the principles identified in this report.

The principles set out in the following chapters aim to help system designers move closer to a complete picture of the interactions in their badge system before issuing any badges. However, they may also be useful to consider as programs evaluate their successes and challenges at any stage.

Acknowledgements

The Design Principles Documentation Project is led by Daniel Hickey and is supported by the MacArthur Foundation’s Digital Media and Learning Initiative. The intended and enacted practices were researched by Dan, Rebecca Itow, Andrea Rehak, Katerina Schenke, and Cathy Tran. Christine Chow and Nate Otto joined the team in 2013, and Nate is currently the project coordinator. Andrea Rehak and Elyse Buffenbarger contributed to the project during its first year. Mimi Ito and An-Me Chung provided crucial input on the initial design of the project. Carla Casilli and Sheryl Grant have provided invaluable input along the way..

References

Casilli, C. (2012, April 27). Badge System Design: standardization, formalization & uniqueness. Retrieved from http://carlacasilli.wordpress.com/2012/04/27/badge-system-design-standardization-formalization-uniqueness/

Cobb, P., Confrey, J., diSessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9–13.

Davidson, C. (2013, July 1). Badges Now. Retrieved from http://www.hastac.org/blogs/cathy-davidson/2013/07/01/badges-now 

Grant, S. & Shawgo, K.E. (2013). Digital Badges: An Annotated Research Bibliography. Retrieved from http://hastac.org/digital-badges-bibliography

Knight, E. (2010, December 27). Certification’ Revisited (#1). Retrieved from http://erinknight.com/post/2488034871/certification-revisited-1