/contribute/signup/ �Task Engagement Test

Date range: February 10 - March 6, 2016

.

Experiment Goals

Summary of Goals as defined in Bug #1168359 (Old bug ~ 8 months old)

• Are users more likely to view information about easy vs. hard tasks?
• Which Ones?
• Are users more likely to complete easy vs. hard tasks?
• Which Ones?
• How does adding a second step to a task impact task completion rate?

​

Test Details: /contribute/

Timeframe:

• February 10 - March 6, 2016

Traffic

• V1: 12,252
• V2: 12,031

Completed Task Definition

• When comparing V1 vs. V2 task completion rates, we’ve used Task Completion Events for the First Step, which is common across both variations.

​

Summary of Findings

• Overall, task completion rates higher across all variation 1 vs. variation 2 tasks.
• The added step may have contributed to a lower task completion rates in v2 vs. v1.
• Significantly more users clicked through to easy vs. hard tasks on the /signup/ page, at 64% vs. 11% respectively - driven by Whimsy.
• Whimsy was the #1 task that users interacted with on the task selection screen (/signup/), and had nearly 10x more clicks than the next highest task - devtoolschallenger.com
• While users were interested in ‘Customizing their Desktop Browser’, only 6% actually installed Whimsy, which was the lowest task completion rate of any task.
• Significantly more users completed hard vs. easy tasks, at 63% vs. 16% respectively - driven by low Whimsy completion rate.
• On V2 pages where we present a tweet button, Avg Intention to Share was at 14% and higher than we historically receive on campaign or product pages (2%).
• This higher share rate may be attributed to the relevance of the page and user’s purpose for being there and wanting to contribute to Mozilla.
• Given these share rates, it would be interesting to know what our total addressable share volume could be?

Summary of Interaction with /signup/

Since both experiences were nearly identical, there was no significant difference in engagement between variations 1 & 2. Engagement with the Whimsy Tile was nearly 10x greater than any other tile.

Task Completion Rates

Task completion rates skew higher among variation 1 tasks.

Task Engagement Detailed Breakdown

Conversion Rates for Whimsy Task

Given the high number of users clicking through on the Whimsy tile, task completion rate was low and directionally(?) higher for v1 vs. v2 at 6.1% vs. 5.3% respectively. 0% of visitors to Variation 2 chose not to engage with the 2nd part of the task.

Variation 1

Variation 2

Conversion Rates for Firefox Mobile Task

​

Variation 1

Variation 2

Conversion Rates for Follow Mozilla Task

Variation 1

Variation 2

Conversion Rates for Joy of Coding Task

Interaction with the Joy of Coding

Completion Video Rate is 40.6% for Variation 1 and 38.6% for Variation 2.

​

Variation 1

Variation 2

Conversion Rates for Build Firefox Task

​

Variation 1

Variation 2

Conversion Rates for Devtools Challenger

Variation 1

Variation 2

Recap: Back to Goals

• Are users more likely to view information about easy vs. hard tasks?
• Yes - Easy. Significantly more users clicked through to easy vs. hard tasks on the /signup/ page, at 64% vs. 11% respectively - driven by Whimsy / Customize Your Browser CTA.
• Are users more likely to complete easy vs. hard tasks?
• Not necessarily. This is really task dependent. Tasks we thought were technically easy to do (ie. Install Whimsy), were the least likely to be completed. Why did only 6% want to install whimsy? Does Whimsy only have limited appeal? How would install rate change if we changed the addon being installed?
• How does adding a second step to a task impact task completion rate?
• Overall, we saw task completion rate directionally lower when an added step was added.

​

Next Steps

• How do we keep users engaged after they completed a task?
• How are we onboarding these users and guiding them down a path to be better quality contributors? What is the next step?
• Email form as a next step
• Should we define potential pathways and tasks for users that want to contribute?
• Consider pushing variation 1 tasks and keeping tasks as simple as possible until we define how we want to leverage this audience.
• Caveat: Share rates were higher than what we typically see. Unless we can drive volume to these tasks, what is the value of these shares and do they generate any substantial traffic back to mozilla.
• Consider testing sharing as a task by itself to Driving Mozilla Advocacy Work.
• Tweets about the Importance of Encryption or Topics such as Net Neutrality could be a lower barrier to contributing in a meaningful way.
• Consider testing other addons to see how that affects task completion.
• Remove Task related to Firefox OS.

Appendix

Unique Pageviews of Task Pages