Machine-assisted article descriptions for Android

Jan 2023

T316375

JT × RS

​

Machine-assisted article descriptions for Android

Why are we doing this?

​

​

​

​

​

​

The Android team is interested in improving the Article Descriptions Suggested Edits quality, while supporting the research of EPFL.

​

In order to provide machine generated article descriptions and have the output of the task show up, a minimum viable product (MVP) for Android needs to be created.

JT × RS

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

The insights from the MVP will be used to:�

• Enhance the algorithm provided by EPFL
• Answer questions about the behavior of our users
• Does it improve quality of edits from new users?
• Does it help users understand what a good article description is?
• Determine the potential success and needed improvements of this project.
• Understanding limitations across languages

​

JT × RS

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

OKRs

Objective: Understand of machine generated article descriptions as an improvement to existing article descriptions while enhancing the article description matching algorithm by developing a MVP

• KR: Establish confidence levels for the the algorithm enough to evaluate language coverage
• KR: User accuracy, stickiness and behavior patterns while using the MVP broken out by contribution tenure exceeds existing feature performance by 5%
• KR: Android team and EPFL has enough information to confidently build or abandon feature
• Guardrail: NSFW reporting is less than 2% and there is no indication of worsening existing bias

​

JT × RS

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

JT × RS

Product Requirements

​

• Integrate with existing system
• In-App Reporting and Feedback mechanisms
• Multilingual compatibility with mBart25
• 1-2 Suggestions per short description
• Ability to edit suggestion

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

JT × RS

​

Machine-assisted article descriptions for Android

Design

​

​

​

​

​

​

JT × RS

Entry point #1:

​

Tapping the ‘Add article description’ CTA in the article

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

JT × RS

Entry point #2:

​

Article descriptions task within ‘Edits’.

​

User needs to be logged-in to see contents of the ‘Edits’ tab.

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

Discovery tooltip?

​

Likely not needed as adding article description task is popular.

JT × RS

Machine-assisted article descriptions for Android

​

​

?

​

​

​

​

​

JT × RS

Onboarding 1/2

​

Opportunity: additional third slide to educate users about machine-assistance.

​

Note: make sure onboarding is shown when entering via Suggested edits.

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

Onboarding 2/2

​

Tooltips point to machine-suggested tags (presented as chips) below the input field.

​

Per Isaac’s recommendation: we display two tags.

JT × RS

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

Placement options for reporting

​

A, B or both?

JT × RS

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

Main screen after onboarding

​

• Keyboard is enabled.
• Chips stay visible and are tappable.

JT × RS

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

Tapping a tag

​

• Fills the input field.
• Hides machine-suggested tags.
• Publish is highlighted
• Tapping the clear icon (x) resets the input field and shows tags again

JT × RS

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

JT × RS

How do chips with tags scroll?

And how do we handle longer suggestions? 👇

​

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

What happens when tapping report?

​

• A form with reasons for the report is shown
• Suggestions that users see are transmitted when form is submitted
• Related Q: Would individual reporting of suggestions make sense?
• Exact contents tbd

JT × RS

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

NO Positive Reinforcement

​

• Out of scope this iteration but if we are having trouble getting enough article description grading per user we will make this addition

JT × RS

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

JT × RS

​

Machine-assisted article descriptions for Android

Experiment & Measurement Strategy

​

​

​

​

​

​

mBART 25

OKRs

KR: Establish confidence levels for the the algorithm enough to evaluate language coverage

• What is the baseline accuracy before and after users are involved?
• Will require some manual grading/patrolling
• How does the accuracy/revert rate change based on editor experience, engagement with feature and language
• How does time spent change with the introduction of feature ?
• How often do users modify the suggestions vs. Add suggestion as is vs. Type in description?
• What is the distribution of responses for reporting and feedback?
• How often is Descartes' first vs. second best suggestion chosen? (Will be randomized for test)

​

JT × RS

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

OKRs

KR: User accuracy, stickiness and behavior patterns while using the MVP broken out by contribution tenure exceeds existing feature performance by 5%

• Evaluate sample of existing article descriptions through the app with and without suggestions compared to staff annotations
• Mean vs. Median length of time to complete tasks by user tenure and response time under 5s
• User returning to task multiple times to complete the task with suggestion and without (1, 2, 7, 14 days)
• How often users are using suggestions, vs. modifying it?
• # of short descriptions completed in a day

​

JT × RS

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

OKRs

KR: Android team and EPFL has enough information to confidently build or abandon feature

• We reach 4000 descriptions complete by at least 2000 unique users
• There are clear answers to other KRs

​

JT × RS

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

OKRs

Guardrail: NSFW reporting is less than 2% and there is no indication of worsening existing bias

​

• Limit to just 2 outputs: the 3rd recommendation is most likely to be problematic (though sometimes 1st/2nd are off too so that's not a panacea).
• Misgendering: determine which languages have gendered forms of words and potentially institute guardrails there (no BLP?) or additional guidance to editors (a "warning" or perhaps the Wikidata-provided gender can be fetched and shown alongside the descriptions though maybe unnecessary if first paragraph is also shown?) The tool could also be limited to only certain access levels of editors too for these types of articles to try to ensure that it doesn't become a moderation headache.
• Ethnicity hallucinations: hard to know what to do about this. Perhaps something similar to gender with no BLP or additional guidance to editors or Wikidata country attributes shown in-screen? But also first paragraph should have useful context too for editors to use in fixing errors.
• Look into model errors with no-lead articles to see if fixable.

​

JT × RS

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

OKRs

Guardrail: Ensure we are not overwhelming patrollers

​

• Stop showing suggestions for a single user in a given day after X # of edits
• Pop up a feedback screen if X# of suggestions are clicked within X# of time

​

JT × RS

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

Qualitative

• Usability tests of the first working version with usertesting.com to optimize and iron out issues will be conducted before production release
• Determine if the interface promotes people understanding the policies around article descriptions and the task overall

Testing Languages:

English, Arabic, Japanese, Hindi, German, French, Spanish and Turkish

JT × RS

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

JT × RS

​

Machine-assisted article descriptions for Android

Open questions

​

​

​

​

​

​

• What if a user keeps selecting the suggestion and putting hitting publish without reading (Quality Gate)
• Will it be fully mobile? Yes -EPFL
• How will we get across that this is scrollable with the chips? Will look at for it for usability testing -EPFL
• Feedback to single suggestions would be helpful (almost vital) - EPFL
• Should keyboard go away each time?
• We need to track the source of recommendations
• Icon to show what is machine generated (do usability testing on this too)
• What happens when it runs out
• Should do randomized order for testing and then do the better one in real implementation

​

​

JT × RS

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​

Questions from design team

​

• Will there be a community consultation with the wikidata community about this feature? Historically I think there is less patrolling on wikidata so would want to ensure they do not get overwhelmed and annoyed to RFC territory.
• Yes, community consultation plan is ongoing (our community relation specialist is tasked with it)
• What “quality gates” or controls could be in place to ensure vandalism or poor edits are limited? For example, a limit to # of MT controls?
• We’re having different control mechanisms in mind:
• This is going to be an experiment, and we’re limiting its exposure
• Limiting the amounts of article descriptions that can be added within a certain time
• We’re simultaneously working on tasks for patrolling that can potentially be linked
• Is there any mechanism to give feedback on bad suggestions and help people explicitly reject them (both so that people understand they don't have to choose poor suggestions and to give feedback that improves the AI if possible)?
• Yes,
• More overall feedback about user expectations - do people realize they are not editing Wikipedia, but are, in fact, editing wikidata? And is this something that would be part of the goal?

JT × RS

Machine-assisted article descriptions for Android

​

​

​

​

​

​

​