Rita Ho & Robin Schönbächler

WIKIMEDIA FOUNDATION / DESIGN

Positive reinforcements to sustain contributions

Cataloguing and reviewing the efficacy of rewards and recognition in sustaining contributions.

​

JUL 2020

Background

INTRODUCTION

A primary objective of the WMF Product team (esp. Growth and Android), is growing the contributor population, concentrating on small-to-medium Wikipedias.

Thus far the focus has been on providing a variety of smaller and more structured tasks to attract new users to start editing, but there has been less experimentation or data available onwiki as to whether different types of positive feedback (thanks, awards, etc) are able to sustain editor contributions, increasing retention for those users who took the first step in making an edit.

​

​

​

WIKIMEDIA FOUNDATION

Goal

INTRODUCTION

Review the various mechanisms that have been employed to encourage people to contribute content to both on and off-wiki products, and assess their relevance and potential impact—if applied—to the retention of newcomers editing on Wikipedia.

​

​

​

​

• Identify and assess the efficacy of different types of positive reinforcement mechanisms used to encourage and sustain user contributions in software.
• Provide recommendations for what mechanisms could be used in the context of editor activation and retention for on wiki products.

WIKIMEDIA FOUNDATION

Hypotheses | Questions

INTRODUCTION

Bestowing users with rewards and recognition that appeal/match to their motivation(s) for editing will encourage contribution.

• Showing users the potential impact of their contributions can instigate their participation.
• Providing positive reinforcement about their involvement will encourage users to continue contributing to Wikipedia.
• Users are more likely to continue contributing when recognition and rewards are personalised and come from a real person.

​

​

• What motivates someone to start editing?
• Do specific messages appeal to specific groups more than others?
• How might the motivating factors to start editing work for sustaining users interests in continuing to contribute?

​

• Are messages of thanks or recognition from real people more effective in motivating contribution over automated sources?
• What types of positive feedback or recognition are more likely to encourage sustained contributions?

WIKIMEDIA FOUNDATION

Approach

INTRODUCTION

1. Identify and evaluable different types of positive reinforcement mechanisms used on:
1. Internal projects and features (Barnstars, WikiLove, Thanks, etc)
2. Relevant offwiki products (e.g., Duolingo, Google Contributions, Fitbit, etc)
2. Literature review of research papers on the subject of positive reinforcement mechanisms to extract any actionable advice (this is effective, that is not, etc.)

3. Provide recommendations on ways to apply the knowledge gleaned from approach 1 and 2 in the form of prioritised taxonomy of positive reinforcement mechanisms to trial.
4. Create initial design explorations for how these ideas could be tested as features to sustain editor retention in Android and Growth products.

WIKIMEDIA FOUNDATION

List of references 1/3

INTRODUCTION

Asadi, Saeid, Shadi Ghafghazi and Hamid Jamali (2013). "Motivating and Discouraging Factors for Wikipedians: The Case Study of Persian Wikipedia". Library Review. 62 (4/5): 237–252.

Bryant, Susan, Andrea Forte and Amy Bruckman (2005). "Becoming Wikipedian: Transformation of Participation in a Collaborative Online Encyclopedia." Proceedings of GROUP: International Conference on Supporting Group Work, Sanibel Island, FL. pp 1-10.

Ciffolilli, Andrea. (2003). Phantom authority, Self-selective recruitment and retention of members in virtual communities: The case of wikipedia. First Monday. 8. 10.5210/fm.v8i12.1108.

Crowston, Kevin & Fagnot, Isabelle. (2017). Stages of Motivation for Contributing User-Generated Content: A Theory and Empirical Test. International Journal of Human-Computer Studies. 109. 10.1016/j.ijhcs.2017.08.005.

Amichai-Hamburger, Yair & Lamdan, Naama & Madiel, Rinat & Hayat, Tsahi. (2008). Personality Characteristics of Wikipedia Members. Cyberpsychology & behavior : the impact of the Internet, multimedia and virtual reality on behavior and society. 11. 679-81. 10.1089/cpb.2007.0225.

Anderson, Ashton & Huttenlocher, Daniel & Kleinberg, Jon & Leskovec, Jure. (2013). Steering user behavior with badges. WWW 2013 - Proceedings of the 22nd International Conference on World Wide Web. 95-106. 10.1145/2488388.2488398.

Antikainen, Maria and Heli Väätäjä (2008). “‘Innovating is Fun’: Motivations to Participate in Online Open Innovation Communities”. ISPIM’08, June 15–18. Tours, France.

Antikainen, Maria & Väätäjä, Heli. (2010). Rewarding in open innovation communities - How to motivate members. International Journal of Entrepreneurship and Innovation Management. 11. 10.1504/IJEIM.2010.032267.

​

WIKIMEDIA FOUNDATION

List of references 2/3

INTRODUCTION

Oreg, Shaul & Nov, Oded. (2008). Exploring Motivations for Contributing to Open Source Initiatives: The Roles of Contribution Context and Personal Values. Computers in Human Behavior. 24. 2055-2073. 10.1016/j.chb.2007.09.007.

Rader, Anna C. (2019-20) - Literature review: Why do people edit?

Rafaeli, Sheizaf and Ariel, Yaron (2008). “Online Motivational Factors: Incentives for Participation and Contribution in Wikipedia”. In A Barak (Ed), Psychological Aspects of Cyberspace: Theory, Research, Applications. Cambridge University Press. Cambridge.

Schroer, Joachim & Hertel, Guido (2009) Voluntary Engagement in an Open Web-Based Encyclopedia: Wikipedians and Why They Do It, Media Psychology, 12:1, 96-120, DOI: 10.1080/15213260802669466

Suzuki, Yoshikazu (2011). Individual and Social Motivations to Contribute to Commons-based Peer Production. MA Thesis. University of Minnesota.

Gallus, Jana (2017) Fostering Public Good Contributions with Symbolic Awards: A Large-Scale Natural Field Experiment at Wikipedia. Management Science 63(12):3999-4015.

Glott, Ruediger, Philipp Schmidt, and Rishab Ghosh. "Wikipedia survey–overview of results." United Nations University: Collaborative Creativity Group 8 (2010): 1158-1178.

Kuznetsov, Stacey. (2006). Motivations of contributors to Wikipedia. SIGCAS Comput. Soc.. 36. 1. 10.1145/1215942.1215943.

Nicolas Jullien. What We Know About Wikipedia: A Review of the Literature Analyzing the Project(s).. 2012, pp.86. ffhal-00857208ff

Menking, Amanda & Rangarajan, Vaibhavi & Gilbert, Michael. (2018). "Sharing small pieces of the world": Increasing and broadening participation in Wikimedia Commons. 1-12. 10.1145/3233391.3233537.

Nov, O., 2007. What motivates wikipedians?. Communications of the ACM, 50(11), pp.60-64.

WIKIMEDIA FOUNDATION

List of references 3/3

INTRODUCTION

Welser, Howard & Cosley, Dan & Kossinets, Gueorgi & Lin, Austin & Dokshin, Fedor & Gay, Geri & Smith, Marc. (2011). 1 Finding social roles in Wikipedia 1. ACM International Conference Proceeding Series. 122-129. 10.1145/1940761.1940778.

Zhang, Xiaoquan, and Feng Zhu. "Intrinsic motivation of open content contributors: The case of Wikipedia." Workshop on Information Systems and Economics. Vol. 10. 2006.

Zhang, Xiaoquan (Michael) and Feng Zhu (2011). “Group Size and Incentives to Contribute: A Natural Experiment at Chinese Wikipedia”. The American Economic Review. 101 (4): 1601–1615.

Zhu, Haiyi & Zhang, Amy & He, Jiping & Kraut, Robert & Kittur, Aniket. (2013). Effects of peer feedback on contribution: A field experiment in Wikipedia. Conference on Human Factors in Computing Systems - Proceedings. 10.1145/2470654.2481311.

​

​

WIKIMEDIA FOUNDATION

1. Research summary and analysis

POSITIVE REINFORCEMENT MECHANISMS

​

​

User motivations & their relationship to positive reinforcement mechanisms

​

​

​

​

1. Research summary and analysis

Research indicates motivations for Wikipedia editors are multifaceted, and shifts over time and experience

​

POSITIVE REINFORCEMENT MECHANISMS

Starter editor

Initially driven by curiosity more than ideological pursuit

​

Episodic Editors, Lapsed editors

Losses from lack of expected support, identity, and community.

Sustained frequent editors

Overcome barriers to develop expertise, status, and identity

Six elements identified as sustaining these veterans

1. Drive content – growing their areas of interest.

2. Make editing a habit – incorporating using Wikipedia as part of their routine

3. Specialize – performing specific tasks (meta, patrol)

4. Identify as Wikipedians – ideologically committed

5. Leverage networks – through building Community relationships and thereby gaining more privileges

6. Cultivate reputations – via creating detailed public user pages to showcase their work, often coupled with receipt of awards such as barnstars and wikilove

Apathy/Discouragement:

The failure of most users to move beyond their first edits at this start is attributed to 3 factors – lack of education or support (not knowing how to do things), lack of identity (not fitting in as a model “Wikipedian”), and lack of community (feeling unwelcome, or else being unaware of a community whatsoever).

These barriers correlates with the NEE field research conducted in 2017.

​

​

Motivation: Curiosity about Wikipedia editability

Wikipedia users to start editing was rooted in curiosity in the editability of Wikipedia. This is either via “gateway edits” to make small fixes, or “organized edits” (e.g., edit-a-thons) for those interested in learning how editing Wikipedia could be useful to them.

WIKIMEDIA FOUNDATION

Starting editor motivations: Curiosity & Social connection

POSITIVE REINFORCEMENT MECHANISMS

If we consider that curiosity about the editability of Wikipedia is the primary driver to first edits, more user education about this fact is an important initial message to disseminate.

Research on online communities indicates people are more likely to start participating if they see that their work will be seen, and in turn engender a sense of connection.[4] Ling et al [5] similarly found people more likely to contribute early on if:

• They believe their effort is important to the group’s performance [Impact]
• They believe their contributions to the group are identifiable [Identity / Recognition]
• They like the group they are working with. [Social]

​

​

Motivation: Curiosity about Wikipedia editability

Wikipedia users to start editing was rooted in curiosity in the editability of Wikipedia. This is either via “gateway edits” to make small fixes, or “organized edits” (e.g., edit-a-thons) for those interested in learning how editing Wikipedia could be useful to them.

​

Starter editor

Initially driven by curiosity more than ideological pursuit

​

​

WIKIMEDIA FOUNDATION

Episodic and lapsed editors: Un-met self-interested incentives

POSITIVE REINFORCEMENT MECHANISMS

One way to frame the barriers that prevent users who start but do not continue to edit is to consider these as unfulfilled motivations for:

• Self-improvement / learning new skills (unfulfilled due to insufficient support and/or education)
• Identity / sense of belonging (discouraged due to perception that similar people are not Wikipedians, esp. those who do not conform to the majority of contributors of a certain gender, race, geography, etc)
• Social connection (discouraged due to lack of welcoming communication, negative feedback, to outright hostility or trolling. Not understanding onwiki communication norms.)

WIKIMEDIA FOUNDATION

Frequent editors: Sustained by common factors

POSITIVE REINFORCEMENT MECHANISMS

The six elements identified as sustaining long-term contributors were:

1. Drive content - currently frequent editors often participate or create WikiProjects or user groups.
2. Edit as a regular activity - frequent editors incorporate perusing community channels (Village pump) and patrolling activities (Watchlists, AFC, etc) as part of their daily routine of logging onto the internet
3. Specialize tasks – seasoned editors who have gained understanding of Wikipedia policies and norms often specialize in ‘advanced’ article quality and oversight tasks, or else perform more bureaucratic roles.
4. Ideological commitment – many long-term editors self-report as being mission-driven, though evidence suggests that individual benefits gained from being users is a larger factor in their commitment.
5. Leverage networks – power users who develop smaller-scale networks are able to collaborative and collective drive individual contributions.
6. Cultivate reputations – within Wikipedia via showcasing their expertise and credibility on self-curated user profile pages.

WIKIMEDIA FOUNDATION

Reverse-engineer sustained editor elements for newcomers

POSITIVE REINFORCEMENT MECHANISMS

For the six elements identified, we may consider designing interventions to make them more accessible for newer users.

1. Drive content
• Short term: Connect users to “recommended WikiProjects” or pair with mentors with shared interests.
• Long-term: explore more structured WikiProjects as “Neighborhoods”
2. Edit as a regular activity - Incorporate activities that can be done more easily while mobile, that is part of everyday internet use (e.g., emails, push notifications, browsing Wikipedia)
3. Specialize tasks – structured tasks, more educational material on participation roles beyond article contributions
4. Ideological commitment – Experiment with different messaging (from the altruistic to more self-interested reasons) when encouraging users to start and continue to participate. This may help us understand in future studies the degree that altruism is a true motivator.
5. Leverage networks
• Introduce more structured user profiles, a place for users to find and network with like-minded individuals.
• Improve onwiki communications (timeliness, usability) so that users can more easily connect
6. Cultivating reputations – structured user profiles including section that show user contributions, awards, conversation, etc in one place.

​

WIKIMEDIA FOUNDATION

Positive reinforcements x Motivations

POSITIVE REINFORCEMENT MECHANISMS

The diagram opposite is a typology of online contribution motivations—based on the literature of Wikipedia editor motivations—for our purposes in mapping to different positive reinforcement mechanisms.

WIKIMEDIA FOUNDATION

Growth team: Positive reinforcements x Retained contributors

POSITIVE REINFORCEMENT MECHANISMS

Activated

New contributor

One successful edit.

Unactivated newcomer

Someone curious to contribute. Indicated by actions like creating an account, selecting to Edit

Lapsed

No contribution after some initial activity.

Active

2+ edits after initial activation.

Re-activated

Returning after a fallow period.

Lost

Never makes another contribution again.

Retained & sustained

Continue long-term editing, often deepening participation in community activities

Each ★ indicates a point of opportunity for positive reinforcement

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

WIKIMEDIA FOUNDATION

Opportunities from reviewing existing onwiki positive reinforcement projects

1. Research summary and analysis

FINDING

Internal projects focus on intrinsic incentives and appeals to altruistic motivations, and are not systematically applied

POSITIVE REINFORCEMENT MECHANISMS

• WikiLove extension
• Barnstars
• Other Wikipedia Awards
• Thanks notifications
• Milestone notifications
• Android (Suggested edits stats, notifications)
• Fundraising
• Growth (Impact module and post-edit dialogs)
• Content translation
• Commons:ISA Tool
• Outreach / WikiEdu
• WikiProjects
• Xtools
• Pageviews

​

Some internal projects/features have lead to minor increases in user activity and retention (e..g, Teahouse, Thanks). However, the types of positive reinforcements used are irregularly applied, in that the award/recognition is typically conferred at the discretion of an experienced editor.[1]

Most internal projects also speak to the assumed altruistic motivations of its contributors (their impact by sharing in the sum of all knowledge).

WIKIMEDIA FOUNDATION

CONSIDERATION

How can discoverability & transparency of rewards be improved?

POSITIVE REINFORCEMENT MECHANISMS

The discretionary nature of gratitude and awards on Wikipedia (e.g., Barnstars, Wikilove, Thanks) means that both conferrers and recipients are mainly established “Wikipedians” already.

Relatedly, they are not systematically used in one way across language wikis or projects (e..g, the WikiLove extension is not on fr, de, cs, or ko).

As such, newcomers likely do not know the existence of such rewards in the first place. For those who do earn an award, understanding of how and why they received it may also be hard to understand due to lack of qualifying criteria in existing rewards.

​

OPPORTUNITY:

How might we consider making such awards more discoverable and understandable?

• Increase prominence of ways for newcomers to give thanks/Wikilove
• Provide a prominent place for existing rewards (e.g., a virtual Barnstar trophy area on a user’s page)
• Add more structure and common rubrics to measure and reward achievements to the existing systems

​

WIKIMEDIA FOUNDATION

CONSIDERATION

Broadening the appeal beyond ideological motivations may improve diversity of retained editors on Wikipedia

POSITIVE REINFORCEMENT MECHANISMS

In a 2017 study, Crowston & Fagnot found evidence to suggest “ideology and group identification plays a lesser role for ordinary contributors than they do for meta- contributors, that is, that Wikipedia acts like a social movement only for contributors at higher levels.” [1]

Furthermore, it seems seasoned editors tend to be of a certain socio-economic status and personality:

“mostly male, characterized by a high level of education and have at least basic computer skills… more likely to have personality characteristics of low agreeableness and high neuroticism, and that introverted rather than extroverted women are more likely to be Wikipedia contributors.” [2]

​

​

​

It is likely that the majority of homogenous editors are retained partly due to existing incentives inadvertently being tailored to attract this group, who can perhaps more “afford” to edit with only intangible, intrinsic rewards.

WIKIMEDIA FOUNDATION

CONSIDERATION

Appealing to more extrinsic and self-focused motivations may also lead to more diversity in people who start editing

POSITIVE REINFORCEMENT MECHANISMS

Non-editor respondents in Wikipedia surveys cite main reasons for not editing because they feel insufficiently knowledgeable, but also because they do not have the time or don’t know how. [1]

This was also noted in a 2016 WMF study of “New Readers” in emerging markets India and Nigeria also found many users lower understanding of how Wikipedia worked (and that it could be edited). [2]

OPPORTUNITY:

Using Fundraising’s messaging testing as a model, there is an opportunity to create user education and onboarding calls to action that appeal to self-focused motivations like learning a new skill or the social connection and fun of group participation.

​

WIKIMEDIA FOUNDATION

CONSIDERATION

Positive messages from experienced users and mentors is proven effective in short-term retention

POSITIVE REINFORCEMENT MECHANISMS

Multiple studies have found that newcomers’ editing activity is increased (short term) when they receive thanks and other positive messages about their work from an experienced editor or mentor. [1][2][3][4]

“Choi et al. (2010), in their examination of Wiki Projects showed positive impact of welcome messages, assistance, and constructive criticism on newcomers' edit levels. This finding is also consistent with Musicant et al.'s recommendation to improve the mentoring program by better matching mentor and adoptees on their field of interest, to improve the empowerment mechanisms (Hansen et al., 2009) of this project.” [5]

​

OPPORTUNITY:

• Build off positive findings from previous research to develop a more formal and efficient way for experienced editors to connect and communicate positively with newcomers.
• Provide different tools to make it easier for newcomers to seek feedback and strengthen relationships with their mentor and other experienced editors.

WIKIMEDIA FOUNDATION

2. Types of positive reinforcement mechanisms

​

​

POSITIVE REINFORCEMENT MECHANISMS

​

​

Badges, Trophies, Medals (Symbolic awards)

POSITIVE REINFORCEMENT MECHANISMS

Description

Symbolic rewards that are discretionary (awarded at the discretion of the conferrer) or ‘threshold’ or merit-based (e.g., reaching X number of contributions of a certain type)

Notes

• Most external sites make use of ex-ante, earn-able awards which has shown effectiveness in increasing user activity. A 2013 paper studied StackOverflow’s badge system which talks about the efficacy especially around the ‘boundaries’ of an award being granted. [1]
• Studies of Barnstars suggest retention increases when it’s seen by the recipient as being from a real person (not a bot), and given a reason for why it was earned.
• Milestone Echo notifications are the closest equivalent to ex-ante rewards onwiki. Their efficacy is limited (though this may be execution and threshold related).

​

​

​

​

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Badges used in apps: (L→ R): Duolingo, Apple Health, Translate Facebook home screen, Translate Facebook badge earned overlay

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Badges used in apps: (L→ R): Google CrowdSource, Google Local Guides Badges, Google Local Guides Support page, Google Local Guides Badge details

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Badges used in apps: (TL→ BR): StackOverflow badge study Tumbleweed, Strava Trophy Case, StackOverflow - a user’s badges page, StackOverflow Help:Badges

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Badges used in apps: (TL→ BR): Khan Academy badge types explained, Badge Showcase on a user page, Topic badge examples

WIKIMEDIA FOUNDATION

Activity streaks (and similar recent activity data)

POSITIVE REINFORCEMENT MECHANISMS

Description

Encourages users to make continued contributions as part of their regular routine (usually daily)

Notes

• Typically personal, on a user stats page
• Often communicated as a push notification with a clear call to action drawing users back to the app to continue their streak
• The effectiveness of this positive reinforcement type corresponds with research which showed “edit as habit” as one of the six main elements that sustained Wikipedia’s frequent editors [1]

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Activity streaks info (top-down, column L→ R): Fitbit steps last 7 days, Fitbit activity this week, , Pokemon Go, StackOverflow, Instagram, Myfitnesspal, Google Local Guides

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Activity streaks info (TL→ BR): Github contribution calendar heatmap, Github activity spider chart, Wikipedia Xtools Time card

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Activity streaks info (top-down, column L→ R): Strava, Nike Run Club Streaks, Nike Run Club streak ‘achievements’

WIKIMEDIA FOUNDATION

Contribution count

POSITIVE REINFORCEMENT MECHANISMS

Description

The number of contributions made by a user, and related statistics such as size or type.

Notes

• Generally appears with other comparative information (e.g., a rank against other users), and impact stats (how many people viewed contributions) to give context to how much recognition or status is deserved
• Special:Contributions as Wikipedia’s main place for this info is unusual in this regard since it does not even offer the count explicitly, but rather a historical list of all contributions by a user
• Xtools.wmflabs.org (stats tool used by experienced editors) could be explored to see what information may be incorporated for new editors

​

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Contribution count info (L→ R): Google Local Guides, Google Crowdsource, Google Translate Community, Strava

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Contribution count info (TL→ BR): Translate Facebook, Foursquare Superuser Tools, Content Translation tool

WIKIMEDIA FOUNDATION

Impact/Interaction/Engagement statistics

​

​

POSITIVE REINFORCEMENT MECHANISMS

Description

Statistics centered around how much others interacted with a user’s contributions, such as retweets or blog views.

Notes

• Often appears with contribution counts so give users a sense of the magnitude of impact (e.g., 5K total views of 500 edits vs 5K total views of 2 edits)
• Wikipedia does not show such stats for individual users and contributions
• Information from https://pageviews.toolforge.org/ offers stats on page interactions (for article and talk pages) that may be mapped to user contributions
• Recent experiments in the Growth team indicate users are interested their “reach” information - with ~30% newcomers are interacting with the “Impact module” which shows pageviews since their edit.

​

​

​

​

​

Add screenshot

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Impact/Engagement info (TL→ BR): TripAdvisor, Google Local Guides, Google Crowdsource, Twitter

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Impact/Engagement info: Medium

WIKIMEDIA FOUNDATION

Leaderboards and Ranking

POSITIVE REINFORCEMENT MECHANISMS

Description

Rankings against others in the contributors, comparisons based on contribution volume, quality, recency, etc.

Notes

• This appeals as a motivation for users seeking to see their contributions in comparison to other contributors for public recognition
• The social component is another important aspect, as it offers a way for users to compete and share the participation experience with their peers
• WhoWroteThat may have potential in extracting more comparative information that could be used to contextualise a user’s contribution as it relates to the larger community of contributors
• WikiProjects are another area where we may enable social connection with users with similar interests

​

​

​

​

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Leaderboard/Rankings TL→BR): Pocket reading app, StackOverflow Reputation score & rank, StackOverflow contributor percentile, Goole Local Guides, Strava

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Leaderboard/Rankings: Strava comparison times, Strava medal board (rank by course)

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Leaderboard/Rankings (L→R): Commons:ISA campaign top contributors, Commons:ISA campaign all stats, Wiki Xtools - Authorship

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Leaderboard/Rankings (L→R): Apple Health trends info, Apple Health sharing, Yelp - other user’s activity, MyFitnessPal community

WIKIMEDIA FOUNDATION

Activity summary (“How you fit in”)

POSITIVE REINFORCEMENT MECHANISMS

Description

Periodical summaries of a user’s activity typically sent as push communications (email, app notifications) to draw users back to using a product.

Notes

• Periodicity tends to be longer (monthly, quarterly, yearly)
• Often presented with comparative metrics (how user’s do X thing compared to an average user, or to a cohort)
• Accumulates other positive reinforcement stats such as longest streak, longest session, most active task, etc
• Framed as a personalised summary of how one fits in
• Splits usage/contributions by THEMES/TYPES
• Sharing to social media is encouraged, which may secondarily promote new user enrolment

​

​

​

​

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Activity summaries (L→R): Fitbit summary email, Spotify Unwrapped (end of year summary)

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Activity summaries: Google Local Guides activity summary emails

WIKIMEDIA FOUNDATION

Levelling up and tier privileges

POSITIVE REINFORCEMENT MECHANISMS

Description

An additional more extrinsic and tangible incentive than badges and counts, whereby users are explicitly shown benefits that may be unlocked by continuing to contribute.

Notes

• Privileges typically offer more ‘insider access’ to more features and/or advanced information
• Access and rights can relate to more control or power over others (e.g., deleting another user’s review)
• Typically the criteria to reach a level and the benefits gained is stated upfront. (This is less true on Wikipedia, where for example on languages with FlaggedRevisions, it is unclear how to gain autoreview status)

​

​

​

​

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Levelling up: Google Local Guides

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Levelling up (L→R): Duolingo, StackOverflow

WIKIMEDIA FOUNDATION

Reactions and rating scores

POSITIVE REINFORCEMENT MECHANISMS

Description

These are counts of other users reactions to a user’s contributions by their selection of generic sentiment feedback mechanisms offered by a product. These reaction scores may be general recognition (Likes, Claps, retweets), or more specific qualitative ratings (Upvotes, Marked as helpful) that are meant to further encourage constructive contributions.

Notes

• Past research on Wikipedia has shown Thanks notifications increase short term editor activity. [1][2]
• Fast one-click feedback from others
• Typically shown publicly
• Counts of quality ratings may relate to other positive reinforcement mechanisms, such as a user’s privileges (e.g, get 50 upvotes to reach next level)

​

​

​

​

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Reactions/Rating scores (TL→BR): Google CrowdSource upvotes score, Yelp reactions, TripAdviser

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Reactions/Rating scores (L→R): LinkedIn, GitHub

WIKIMEDIA FOUNDATION

Personalised praise/gratitude

POSITIVE REINFORCEMENT MECHANISMS

Description

These are messages of thanks or praise intended to provide more specific encouragement to contribute than a reaction and rating.

Notes

• Research has shown that personalised Talk page messages work. However, more experimentation is needed to determine whether praise needs to come from a real person.
• One study (Zhu et al 2013) found that besides Positive feedback, even negative, directive, or social feedback had positive effects on newcomer editing activity. Experienced editors activity was not affected.
• Private thanks or Public testimonials? Another area where efficacy one way or another is unclear.

​

​

​

Add screenshot

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Personalised praise (clockwise from top): LinkedIn, GitHub, User:Talkpage message (potentially added via Wikilove extension), Wikilove extension send message UI

WIKIMEDIA FOUNDATION

Certificates and other official/professional recognition

POSITIVE REINFORCEMENT MECHANISMS

Description

Verified ticks or other certification that can used to bolster a user’s professional or educational credentials, typically to certify expertise or competence in a certain skill/topic area.

Notes

• This is akin to a badge or other milestone award, but the emphasis on the user passing a formal assessment.
• The submission of articles to be a featured article or images to Commons for Image of the day can be seen as a close Wikimedia proxy to seeking certification
• On WikiEdu, users can complete edit tutorials to get virtual certificates of completion.

​

​

​

​

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Certificates/Professional recognition: screenshots of LinkedIn’sSkill Assessement tool

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Certificates/Professional recognition (L→R): GitHub, StackOverflow

WIKIMEDIA FOUNDATION

Financial/Material rewards

POSITIVE REINFORCEMENT MECHANISMS

Description

Money, gift cards, merchandise giveaways, and other tangible rewards that have attributable dollar value. Contributions are made more transactional, and require more guardrails to ensure quality is maintained.

Notes

• Per Antikainen & Vaataja (2008) - monetary incentives in online communities like Amazon Mechanical turk are effective in generating contributions of good quality, but require careful design.
• Wikimedia projects rarely include material rewards in online events, but edit-a-thons will often include small merchandise giveaways (stickers, stationery, etc). They work much more like ex-post rewards here in that they are less expected and not a guaranteed transaction.

​

WIKIMEDIA FOUNDATION

POSITIVE REINFORCEMENT MECHANISMS

Examples of Financial/Material compensation (L→R): Spare5, Amazon Mechanical Turk

WIKIMEDIA FOUNDATION

3. Design considerations

POSITIVE REINFORCEMENT MECHANISMS

​

​

CONSIDERATION

High quality, constructive contributions over mere activity

POSITIVE REINFORCEMENT MECHANISMS

Qualitative ratings are used in external products to help control the quality of contributions – either positive (e.g., X users found your review helpful on Google Local Guides), and/or negative (e.g., # thumbs downs on a StackOverflow answer).

Wikipedia mainly controls quality by the drastic step of reverting newcomer’s contributions. We may consider how new positive reinforcement types could be introduced to that more positively affirm to users to edit constructively, such as to recognize and assess contributions by quality (e.g., not reverted, number of thanks, re-use of a citation added) before the negative message.�

​

WIKIMEDIA FOUNDATION

CONSIDERATION

Emphasizing Community and public recognition

POSITIVE REINFORCEMENT MECHANISMS

In contrast to many external products where users are able to share publicly the ‘proof’ of their achievements (e.g., LinkedIn expertise, GitHub user profiles), Wikipedia recognition, achievements, and/or status is either personal or else shared with “insiders” of a particular wiki community sub-group.

Since newcomers are typically unaware of their existence in a community, a design consideration for any new recognition system would likely benefit from incorporating more introduction and connection to the community of editors to which they have joined.

WIKIMEDIA FOUNDATION

CONSIDERATION

Push communications to users

POSITIVE REINFORCEMENT MECHANISMS

It’s commonplace for external products to reach users off-product via email and app push notifications.

On Wikipedia, users must go to the project to see Echo notifications, and go to their specific User talk page to see thanks or awards. However, there are two internal cases of push communications that have been effective:

1. Fundraising emails: In its first year (2011), emails accounted for 6% of all donations, and increased almost each year to most recently being 26% in FY2018/19. [1]

2. Android local notifications: 3-day notifications increased edits by 71%, whilst 7-day notifications increased 24-day retention by 4.8%, and edits by 98%.[2]

WIKIMEDIA FOUNDATION

CONSIDERATION

Specific numbers and details matter in impact messaging

POSITIVE REINFORCEMENT MECHANISMS

External products often convey positive reinforcement messages using specific metrics (e.g., X number of reviews) and comparison figures (top Y% of users).

Within Wikipedia, including specific details has proven effective in increasing engagement in the Fundraising context. For example, A/B tests to include details such as the day of the week and a user’s location (This Monday in Copenhagen, make a donation of DKK 50...) in a donation banner or email leads have shown higher donation rates.

WIKIMEDIA FOUNDATION

CONSIDERATION

Topics and common interests as key social motivators

POSITIVE REINFORCEMENT MECHANISMS

Categorizing contributions by topics of interest (Design, Music, Physics, etc) or task types (e.g, Level X Restaurant reviewer, Top 1% image contributor) provide users with inducements to continue that appeal directly to their sense of belonging to a certain social group, or professional standing.

Internally, increase in editing activity has been seen in the Content Translation tool and Growth’s Newcomer tasks when users were provided with the ability to get suggestions based on their chosen areas of interest.

WIKIMEDIA FOUNDATION

CONSIDERATION

Newcomer-focused or extensible for all contributor tenures

POSITIVE REINFORCEMENT MECHANISMS

Should a reward/recognition system be relevant for all users?

Certain features such as the pageview stats in Growth’s Newcomer Impact module have anecdotally been requested by some experienced editors who signed on as mentors.

On the other hand, other rewards/ recognition types are less likely to have an effect on experienced users. For example, milestone badges may have diminishing returns of impact.

In a 2013 study where feedback messages were sent to editors, whilst newcomers increased editing activity, the same messages had no such effect on experienced users; with some of those users even perceiving them negatively as challenges their expertise. [1]

​

WIKIMEDIA FOUNDATION

CONSIDERATION

The importance of joyful design details

POSITIVE REINFORCEMENT MECHANISMS

Visual details and animations are an important part of the execution/delivery of positive reinforcement messages. The is seen for example in the “delightful” animations which act like transitory celebrations when an activity is complete in apps like Duolingo.

The “fun factor” is also part of user motivations from users surveyed onwiki [1][2], and in other online open source communities [3].

WIKIMEDIA FOUNDATION

Appendix

POSITIVE REINFORCEMENT MECHANISMS

Reviewed internal projects/features

APPENDIX A

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Barn stars 1/2

References

1. Wikipedia:Barnstars
2. Wikipedia talk:WikiProject Wikipedia Awards
3. Wikipedia:Barnstars 2.0/Guidelines
4. Experimental Study of Informal Rewards in Peer Production
5. Meta Research:Newsletter review 2014/07
6. https://pubsonline.informs.org/doi/pdf/10.1287/mnsc.2016.2540

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Barn stars 2/2

Screenshot of some Wikimedia barnstars from https://commons.wikimedia.org/wiki/Category:Wikimedia_barnstars

References

• https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Wikipedia_Awards

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Wikipedia awards

References

• Extension:Thanks - MediaWiki
• Meta:Understanding Thanks research project
• Thanks for the thanks! Examining the efficacy of Wikipedia’s thanks feature
• https://arxiv.org/pdf/1903.03308
• Thanks log - Wikipedia
• Citizensandtech.org - Jun 2020 Thanks research
• Citizensandtech.org - Jun 2020 Mentor study

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Thanks notifications

References

• Help:Notifications/Types
• Mediawiki: Notifications
• https://phabricator.wikimedia.org/T124003

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Milestone notifications

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

WikiLove 1/2

References

• Wikipedia:WikiLove
• WikiLove extension
• Wikiapiary - WikiLove usage
• WikiLove/Design
• Research:WikiLove
• WikiLove/Analysis

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

WikiLove 2/2

Top: Screenshot showing the heart icon triggering the WikiLove form

Bottom: Send WikiLove form

Results from a research study on usage on enwiki (Research:WikiLove)

While the two heatmaps illustrate that senders tended to be slightly younger users (by year of registration), the pie chart shows that 80% of senders were by established Wikipedians (those with >100 edits). Finally the content analysis showed a quarter of messages were not praise or thanks.

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Growth: Newcomer homepage Impact module

​

References

• https://www.mediawiki.org/wiki/Growth/Personalized_first_day/Newcomer_homepage#Impact_module
• https://docs.google.com/spreadsheets/d/18Pw0PdPzEdNcQFehVt03jWrwoi6M9oXhQFjUJqgjzp4/

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Growth: Newcomer tasks - Onboarding messages 1/2

References

• Newcomer Tasks – Variant Test 1 Results (A vs. B)

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Growth: Newcomer tasks - Onboarding messages 2/2

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Growth: Newcomer tasks - Post-edit engagement

References

• ⚓ T245790 Newcomer tasks: post-edit dialog

References

• Product's July 2020 Quarter-in-Review
• Superset: AndroidNotificationABTest/
• 2018 welcome notifications edit impact

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Android: Suggested edits local notifications

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Android: Suggested edits user stats (1/3)

References

• Impact of Suggested edits V3 (RS) - Google Sheets
• Suggested edits V3 (slides)
• Suggested edits V3 (doc)
• Suggested edits monthly analytics

Daily active usage before and after the introduction of profile stats in November 2019 (Source).

Disclaimer: Profile stats have been rolled out together with eligibility criteria changes of the feature. Suggested edits has been made accessible to all logged-in users, whereas it needed to be unlocked first. No statistically significant data for the isolated intro of profile stats is available.

​

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

​

Android: Suggested edits user stats (2/3)

​

Eligibility changes showing Suggested edits to all logged in users combined with the introduction of profile stats 3 months later produced a ± 10x uplift in editor numbers (Source)

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Android: Suggested edits user stats (3/3)

References

• Product Oct 2020 Quarter-in-Review
• Android User Contributions Interaction data
• Usability test findings from May 2020
• Suggested edits V5 on Phabricator

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

​

Banner Fundraising

Images:

Top: Design 1 adopted as control banner

Bottom: Design 2 - one of the CTAs with no effect

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

​

Email Fundraising 1/2

References

• Positive reinforcements in email fundraising - Google Sheets
• Wikimedia Email Stats 2018
• Wikimedia Email Stats 2019

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Email Fundraising 2/2

References

• https://commons.wikimedia.org/wiki/Commons:ISA_Tool
• https://commons.wikimedia.org/wiki/Commons:ISA_Tool/Challenges

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Commons:ISA tool 1/2

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Commons:ISA tool 2/2

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Content translation 1/2

References

• MW: Content translation
• MW: Content translation/Boost
• MW: ...ContentTranslationStats

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Content translation 2/2

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Wiki Education / Outreach Dashboard 1/3

References

• https://wikiedu.org
• https://dashboard.wikiedu.org/explore
• Outreachdashboard.wmflabs.org

​

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Wiki Education / Outreach Dashboard 2/3

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Wiki Education / Outreach Dashboard 3/3

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Event Metrics (f.k.a Grant Metrics) dashboard 1/2

References

• https://meta.wikimedia.org/wiki/Event_Metrics

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Event Metrics (f.k.a Grant Metrics) dashboard 2/2

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

WikiProject project pages 1/2

References

• WikiProject
• https://en.wikipedia.org/wiki/Wikipedia:WikiProject_Women_in_Red

Article alerts shows active discussions taking place

# participants shown helps promote the ‘liveliness’ of the group

Any user can change any other user’s template profile information

List of members to promote community, but unclear what interactions are encouraged

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

WikiProject project pages 2/2

Strange double navigation concept - the project home page loads with all sections/modules shown in a full page, but selecting to view one section in the nav menu reloads just that section of the project.

Metrics solely focuses on Article creations

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Xtools 1/3

References

• XTools 3.10.16 documentation
• Admin Score
• 2. API — XTools 3.10.16 documentation

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Xtools 2/3

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Xtools 3/3

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Pageviews 1/2

References

• https://meta.wikimedia.org/wiki/Pageviews_Analysis

APPENDIX: REVIEWED INTERNAL PROJECTS/FEATURES

Pageviews 2/2

Reviewed external products

​

​

APPENDIX B

Previously reviewed products from this comparative review

APPENDIX B: REVIEWED EXTERNAL PRODUCTS

Refer to the appendix in our prior comparative review for reviews of the following:

• Google Crowdsource
• Google Local Guides
• Duolingo
• Google Translate Community
• Translate Facebook
• Foursquare Superuser tools
• MyFitnessPal
• Amazon Mechanical Turk, Spare5
• LinkedIn
• Apple Watch
• Vivino

• Twitter
• Instagram
• Yelp
• IMDB
• TripAdvisor

WIKIMEDIA FOUNDATION

APPENDIX: REVIEWED PRODUCTS

Stack Overflow

STACK OVERFLOW

Caption

STACK OVERFLOW

Caption

STACK OVERFLOW

Caption

STACK OVERFLOW

Caption

APPENDIX: REVIEWED PRODUCTS

GitHub

GITHUB

Different elements from GitHub’s user profile and project pages

GITHUB

Insights/Project stats

GITHUB

New Profile README examples

GITHUB

Group collaboration feedback features

APPENDIX: REVIEWED PRODUCTS

Khan Academy

KHAN ACADEMY

Caption

KHAN ACADEMY

Caption

KHAN ACADEMY

Caption