1 of 36

Mentoring new editors

on Wikipedia

Kirsten Stoller

WMF Product Manager, Growth

Martin Urbanec

WMF Senior Software Engineer, Growth

2 of 36

Speakers

Wednesday, August 16 (14:15-15:15)

Kirsten Stoller

WMF Growth team Senior Product Manager

Martin Urbanec

WMF Growth team

Senior Software Engineer

3 of 36

Agenda

Presentation: ~15 minutes

New editor needs
Mentorship
Mentorship metrics

Q & A: ~ 10 minutes

Workshop: ~35 minutes

Small group discussions

4 of 36

Why are we here?

We care about the sustainability of Wikipedia, and are concerned about newcomer retention.

The majority of new accounts never edit.
The majority of first time editors never return to edit again.

5 of 36

Why are you here?

Raise your hand if you help newcomers or provide mentorship is some way within the Wikimedia movement.

6 of 36

Why are you here?

Raise your hand if you are currently a Mentor using Growth’s mentorship tools on Wikipedia.

7 of 36

Is Mentorship available at my Wiki?

The following wikis have Growth’s Mentorship features enabled:

Albanian Wikipedia

Arabic Wikipedia

Armenian Wikipedia

Azerbaijani Wikipedia

Basque Wikipedia

Bengali Wikipedia

Central Kurdish Wikipedia

Chinese Wikipedia

Croatian Wikipedia

Czech Wikipedia

Dutch Wikipedia

English Wikipedia*

French Wikipedia

French Wiktionary

German Wikipedia

Greek Wikipedia

Hebrew Wikipedia

Hungarian Wikipedia

Indonesian Wikipedia

Italian Wikipedia

Kannada Wikipedia

Kashmiri Wikipedia

Korean Wikipedia

Kurdish Wikipedia

Latvian Wikipedia

Moroccan Arabic Wikipedia

Nepali Wikipedia

Northern Frisian Wikipedia

Northern Sami Wikipedia

Norwegian Bokmål Wikipedia

Polish Wikipedia

Portuguese Wikipedia

Romanian Wikipedia

Russian Wikipedia

Serbian Wikipedia

Slovak Wikipedia

Spanish Wikipedia

Swahili Wikipedia

Swedish Wikipedia

Telugu Wikipedia

Test Wikipedia

Tumbuka Wikipedia

Turkish Wikipedia

Ukrainian Wikipedia

Vietnamese Wikipedia

Romanian Wikipedia

Russian Wikipedia

Serbian Wikipedia

Slovak Wikipedia

Spanish Wikipedia*

Swahili Wikipedia

Swedish Wikipedia

Telugu Wikipedia

Test Wikipedia

Tumbuka Wikipedia

Turkish Wikipedia

Ukrainian Wikipedia

Vietnamese Wikipedia

~50 wikis

~800 Mentors

Growth’s Mentorship features are available at all Wikipedias, but require a local admin to enable it, and have at least one mentor signed up. So in case you aren’t sure if your local wiki has Growth’s mentorship enabled, here’s the complete list. Also, notice that Mentorship is available at Test Wikipedia, so you have a safe testing environment to give it a try!

That’s about 50 wikis with approximately 800 mentors in total (mentors on multiple wikis are counted twice).

If you are wondering about the asterisk next to English and Spanish wikipedia both have Mentorship enabled, but not enough mentors to scale the feature to ALL new accounts created. On all other wikis, 100% of new accounts are paired up with a Mentor.

Growth’s mentorship is mainly aimed at Wikipedia, but notice French Wiktionary on the list… so the feature is useful outside of Wikipedia.

8 of 36

Editing is complicated for newcomers

New editors face three barriers when getting started

(New_Editor_Experiences)

“What is an infobox?”

Technical

“What is notability?”

Conceptual

“Why are people so mean?”

Cultural

9 of 36

What do new editors need?

New editor needs:	Growth team tool
In-context help: help relevant to the activity they are doing at that moment.	Help panel
Human-to-human help: One-on-one help from an experienced community member.	Mentorship
Task recommendations: suggestions for specific tasks based on their interests and skills.	Newcomer homepage Suggested edits

New Editor Experiences, August 2017 | By Reboot and the Wikimedia Foundation

10 of 36

Mentorship

Hypotheses:

If we improve mentorship tools, then mentors can provide a better experience for newcomers.

If newcomers have a better experience, then new editor retention will improve.

New Wikipedia editors, Molly Fuller Abbott, CC BY-SA 4.0, via Wikimedia Commons

11 of 36

Mentorship

Mentorship tools are available at all Wikipedias, you might just need to set it up.

Experienced wikipedians volunteer as Wikipedia mentors

Every mentor gets assigned newcomers (mentees) randomly

Enrolling: Special:EnrollAsMentor
Enabling: Special:EditGrowthConfig

Special:ManageMentors page at English Wikipedia

Special:EnrollAsMentor at Test Wikipedia

12 of 36

Mentorship

Admins can edit mentorship options of any user, or fully remove mentors.

Upon mentor removal, mentees are reassigned to a different mentor.

Admin interface for editing another mentor

Admin interface for removing another mentor

13 of 36

Mentor Dashboard

A dashboard for experienced users who signed up to become mentors to newcomers.

Mentors to have a central place to access tools and settings for mentoring newcomers.

It allows mentors opportunities to proactively reach out to help newcomers.

Mentor Dashboard

More info at MW:Mentor_dashboard

14 of 36

Mentors encourage newcomers

Hypothesis:

If Mentors proactively send new editors supportive messages, new editors will be more likely to continue editing.

Plan:

One part of the Positive Reinforcement project was to add a new module to the Mentor dashboard and send mentors notifications when they have “praise-worthy” mentees.

15 of 36

Thoughts from Mentors:

“Mentors are the first guard of the wiki, avoiding many of those users to create bad content that would need to be deleted otherwise. So, in theory, mentors are reducing the workload of patrollers and admins.”

-Spanish Wikipedia Mentor

Pat Hadley guides new Wikipedia editors, Molly Fuller Abbott, CC BY-SA 4.0, via Wikimedia Commons

Before I share the recent mentorship metrics, I wanted to share this thought from a current mentor on Spanish Wikipedia.

“Mentors are the first guard of the wiki, avoiding many of those users to create bad content that would need to be deleted otherwise. So, in theory, mentors are reducing the workload of patrollers and admins.”

This is an important consideration when we look at qualitative data on Mentorship because Mentorship can mean many different things, and the reality is that sometimes Mentors have the role of discouraging newcomers from writing that COI article or writing an article on a topic that doesn’t meet Notability requirements.. Hopefully mentors can help funnel some of these newcomers into productive ways to contribute, but I want to acknowledge that sometimes Mentors are providing guidance that might actually discourage newcomers from completing the specific edit or task that they drove them to create an account. So it’s really challenging to clearly evaluate the impact of Mentorship in a quantitative way.

16 of 36

Mentorship Metrics

A WMF data scientist gathered registrations from Spanish and English Wikipedia from June 1 to July 15, 2023. English and Spanish Wikipedia don’t yet provide Mentorship to 100% of new accounts, and since mentorship is randomly assigned this is a natural A/B test.

Our dataset contains 19,305 Spanish accounts and 114,512 English accounts, of which 50.0% and 11.2% respectively had Mentorship enabled.

Further details:

mediawiki.org/wiki/Growth/Positive_reinforcement/Mentorship_preliminary_analysis,_August_2023

17 of 36

Mentorship Metrics

What proportion of newcomers with access to a mentor, ask them a question?

18 of 36

Mentorship Metrics

What proportion of newcomers with access to a mentor, ask them a question?

19 of 36

Mentorship Metrics

Is asking their mentor a question their first edit?

20 of 36

Mentorship Metrics

Is asking their mentor a question their first edit?

21 of 36

Mentorship Metrics

Does mentorship increase the number of newcomers who edit for the first time?

22 of 36

Mentorship Metrics

Does mentorship increase the number of newcomers who edit for the first time?

Activation: making an edit within 24 hours of registration.

Registration platform

No Mentorship

With Mentorship

23 of 36

Mentorship Metrics

How does Mentorship impact retention, productivity, and revert rates?

Retention: an activated newcomer returns to edit again within 14 days.
Productivity: the number of edits made.
Revert rate: for newcomers who make at least one edit, the proportion of their edits that were reverted within 48 hours.

24 of 36

Mentorship Metrics

How does Mentorship impact retention, productivity, and revert rates?

Retention

We find no difference in the retention of newcomers who have access to a mentor compared to those who do not, when keeping their first day activity constant.

Productivity

We find conflicting patterns of productivity between English and Spanish Wikipedia.

Revert rate

We find no difference in the revert rate between newcomers who have access to a mentor and those who do not, when keeping their activity constant.

Further details:

mediawiki.org/wiki/Growth/Positive_reinforcement/Mentorship_preliminary_analysis,_August_2023

Once again it’s never easy to provide a clear yes or no, as results vary by wiki, registration platform, etc.

Retention

We find no difference in the retention of newcomers who have access to a mentor compared to those who do not, when keeping their first day activity constant. This means that for two newcomers who registered on the same wiki, using the same platform, and who made the same number of edits on their first day, the likelihood of them sticking around and editing again is the same.

That being said… it’s worth noting that only about 2% of newcomers ask questions… but we can’t compare that to the equivalent 2% that might have asked questions had they had mentorship. So we are comparing ALL new accounts that have access to Mentorship vs. ALL accounts that do not have access. In other words, it’s possible that for the 2% who are interacting with mentors, that there is a shift in some of these metrics, but it’s not significant enough to shift the metrics as a whole.

Our data scientist attempted to dig into this further with a two-stage least squares analysis, but that further analysis showed conflicting results. Admittedly I’m still digesting this report and considering the implications, and honestly it leaves me with more questions than answers. One major one being: how can we transition mentorship into being not just a Q&A service but also about cultivating relationships.

Productivity

We find conflicting patterns of productivity between English and Spanish Wikipedia. On English Wikipedia, we find an increase in productivity for newcomers who registered on desktop, and no difference for those who registered on mobile web. When it comes to Spanish Wikipedia, desktop users show no difference, whereas mobile web users see a decrease in productivity when they have access to a mentor.

Revert rate

We find no difference in the revert rate between newcomers who have access to a mentor and those who do not, when keeping their activity constant.

25 of 36

Mentorship Metrics

It’s hard to really impact editor retention!

Active editors on English Wikipedia

Active editors on Spanish Wikipedia

Graphs from stats.wikimedia.org

In summary it seems like mentorship might help new accounts edit for the first time, but we didn’t see the shift in retention that we had originally hoped for, but it’s REALLY hard to change user behavior and impact editor retention.

And like I mentioned previously, it’s challenging to look at just the quantitative metrics: Mentorship is providing benefits that don’t always result in more edits. And the needs and impact of new features or programs can vary significantly across wikis!

Furthermore, mentorship can be far more powerful if Mentors engage with newcomers in more meaningful ways than just answering questions. I’m still very hopeful that the new Positive Reinforcement / personalized praise tools will meaningfully extend mentorship beyond how it’s currently used on many wikis.

26 of 36

Thoughts from Mentors:

“I truly believe that the combination between virtual mentorship and in-person guidance could make the difference. At my user group we add all the attendants of our workshops as mentees… they feel relief to see our username in the welcome page.”

-Spanish Wikipedia Mentor & Wikiesfera organizer

PatriHorrillo, CC BY-SA 4.0 via Wikimedia Commons

27 of 36

Sign up!

It’s easy! Simply visit Special:EnrollAsMentor

on your home wiki.

28 of 36

Set up Mentorship at your Wiki!

1. Agree on having mentorship: Start a conversation at the relevant places at your wiki & recruit mentors.

2. Activate mentorship: At your wiki, an interface admin can turn on the mentorship system, via Special:EditGrowthConfig. Add some criteria to define who can be a mentor.

3. Mentors sign up

Editors can sign up to be Mentor: Special:EnrollAsMentor

New account holders see a message from their Mentor on their Homepage

29 of 36

Q & A

Do you have any questions?

30 of 36

Discussion

What is one thing your wiki does well in terms of welcoming or onboarding newcomers?
How can we recruit more Mentors and ensure Mentorship is rewarding and engaging for Mentors?
How can we make Mentorship more effective? How should Mentorship tools be improved?

Independent brainstorm: Spend 5 minutes adding sticky note answers or ideas in response to questions

Add your ideas (sticky notes) to the associated poster

Discussion groups: pick a topic you want to discuss in more detail.

31 of 36

Discussion feedback

What is one thing your wiki does well in terms of welcoming or onboarding newcomers?

Welcome templates on user talk pages, barnstars on user talk pages
Letting know that “abandon hope all ye enter here” :)
Welcome templates, communication through social media (Twitter, Discord, etc.)
Having fun training sessions frequently
Assigned mentors choose new mentors
Every mentor has 1-2 co-mentors
Nothing special
Mentors are very committed
Use live meetings of Wikipedians Germany: stammtische, wikicon, GLAM to recruit them
Welcome messages for newbie with guides

32 of 36

Discussion feedback

How can we recruit more Mentors and ensure Mentorship is rewarding and engaging for Mentors?

Advertise on village pumps and showcase how easy / not overbearing mentorship is
Measure mentorship efficiency and give the best performers some gadget (hat, wikibiscuits,....) And also a bandstar or something similar
Move sessions like this / more time / more exchange amongst mentors
Live mentor meetings in countries / worldwide
By training them better, giving them more tools
Auto-enroll as a mentor? Prompt it in more places
Posting asking for mentors to sign up in key places: village pumps, The signpost, wikiproject talk pages, watchlist notices
Mentor ambassadors → who are super mentors?
Not all may be aware of the Mentorship program. Maybe prompt users to sign up upon logging in? Much like notifying them about ongoing requests for adminship & suchlike.
Instead of randomly assigning mentors, connect people who have shared interests (e.g. subject or location)
Recruit work with community members local affiliates to spread awareness
German wikipedia: there is a functioning system (at least I believe so since I am not one) It is an important function for newbies.
Offer awards, tokens of recognition to active members (those who answer quickly)
A workshop for mentors? Some recognition for mentors?
To see contributions of their mentees
Trainings for new mentors who may feel they’re not ready to work with a person (instead of doing cleanup)

33 of 36

Discussion feedback

How can we make Mentorship more effective? How should Mentorship tools be improved?

New users should receive a traditional mail with info because may have no experience to find info in wikipages
Train them find ways to evaluate their individual efficiency
More practical small tasks like adding links
Organize often mentors (?)
Notifications w/ a thank you for successfully helping 1/10/100 newcomers
Create tools for identifying inactive members and suggest them for removal
Less is more. Make it as simple as you can. Probably start off by guiding the newbie with correcting typos, etc.
Coach-the-coach style support for mentors
Make it easy to pass questions to other mentors (ticketing system) if delay in response or specialist needed
Tool that shows (highlights) new comments from mentor on discussion pages
Tool: survey for each newbie on how his level of tech-knowledge is (as this impacts the answer you will give to newbies)
Create community
Tool box with my mentors personal answers to easily insert + adapt
Maybe include a space for mentors discussions in order to streamline and have community-wide standards

34 of 36

Thank you!

Are you interested in becoming a Mentor or setting up Mentorship?

Learn more: mediawiki.org/wiki/Growth

35 of 36

36 of 36

Appendix: Mentorship Metrics

Methodology

We gathered a dataset of registrations from English and Spanish Wikipedia spanning June 1 to July 15 2023. During this time period, 50% of newcomers on Spanish Wikipedia were randomly assigned to have access to a mentor, while the other 50% did not get a mentor. On English Wikipedia, 10% of newcomers would randomly get a mentor until July 11, at which point it increased to 25% (ref).

Our data gathering excludes known test accounts, bots, users registered through Wikipedia's API (these are usually mobile app accounts), and users not registered on the given wiki (i.e. autocreated accounts and those registered by another user). We also excluded users with a non-standard mentorship setting (there were a handful of these on each wiki).

The dataset used in this analysis contains 19,305 Spanish accounts and 114,512 English accounts, of which 50.0% and 11.2% respectively had Mentorship enabled.

In the above analysis of mentorship's effect on the key Growth metrics of activation, retention, productivity, and revert rate, we make extensive use of regression models. Activation and retention are yes/no outcomes, for which we use a logistic regression model. Productivity is measured by a count of the number of edits, which is known to have a long tail distribution. We therefore use a negative binomial model for productivity. Our revert rate analysis uses a zero-one-inflated beta distribution. This is because revert rates calculated across a time window tends to fall into one of three categories: 1) the user has all of their edits reverted (one-inflation), 2) the user has none of their edits reverted (zero-inflation), and 3) the user has some of their edits reverted (resulting in a beta distribution).

Further details:

mediawiki.org/wiki/Growth/Positive_reinforcement/Mentorship_preliminary_analysis,_August_2023