1 of 21

Activity Stream

Go/No-Go Decision Meeting

2 of 21

Agenda

Review Shield studies and what we’ve learned
Contingency plan and dependencies
Options for shipping in 57
Next Steps

3 of 21

56 Release Shield Study

Week 1 Retention

Activity Stream: 81.81%

Control: 92.11%

Weekly Cohort Retention Chart

https://sql.telemetry.mozilla.org/queries/47860/source#129378

Created By: Peter Dolanjski

Start Date: 2017-10-04

End Date: 2017-10-22

Total Users: 1,410,584

Control Users: 50% of total users

Variant Users: 50% of total users

This is a weekly cohort analysis of retention for the 56 Release Shield study. The Start_Week column represents the week the cohort started, followed by the branch, followed by the rate they returned each week that follows. In this chart new and existing users are grouped together. New users are ones who have profile creation dates newer than when the experiment started.

You can see that there is 10+ point drop in week 1 retention (yellow box).

We were surprised to see that there was such a significant drop in Control group retention for each subsequent cohort’s first week. This was the first indication that something might be wrong with the data.

4 of 21

56 Release Shield Study

Existing users

New users

Daily Stacked Retention

https://sql.telemetry.mozilla.org/queries/48488?p_start_date=2017-10-04&p_experiment_id=pref-flip-activity-stream-56-release-bug-1405332&p_end_date=2017-10-22#130853

5 of 21

56 Release Shield Study

Activity Stream

Control

Activity Stream

Control

Firefox Retention vs Experiment Retention

https://docs.google.com/spreadsheets/d/1yE9c6jcUUaAPdRIhh2K_twzi6bZq3x1D7KyLSArTFY8/edit#gid=1719365146

Created By: Rob Rayborn

Start Date: 2017-10-04

End Date: 2017-10-22

Total Users: 1,410,584

Control Users: 50% of total users

Variant Users: 50% of total users

Rob Rayborn, from S&I, has been looking into this data over the past 2 weeks. He has been working to compare the retention that we see in Firefox overall with the retention we see from the experiments dataset. He pulled together a preliminary analysis which he was able to share with us on Friday. This is still being reviewed by folks on Katie’s team, but it’s very helpful to explain what might be going on.

Firefox Retention: Retention as measured from clients_daily dataset. This tells us how many people came back to Firefox.

Experiment Retention: Retention as measured from the experiments dataset. This ultimately tells us how many people remained in the experiment.

We see the same 10 point drop in week 1 from the experiments table that we saw in the first slide (yellow box). We do not see the same drop in Firefox retention (orange box).

While we can’t use the Firefox retention data here as valid for this experiment (since people dropped out at odd times), we can see that there wasn’t a large drop in retention. By the second week both groups have the same retention numbers.

We aren’t sure at this point why users dropped out of the experiment. S&I is working with the data pipeline team to debug this.

Based on this analysis, we don’t see the 10 point retention drop in the Firefox retention data that we saw in the experiment data.

6 of 21

57 Beta Shield Study

Existing users

New users

Daily Stacked Retention

https://sql.telemetry.mozilla.org/queries/48366?p_experiment_id=pref-flip-activity-stream-57-beta-enabled-bug-1410535&p_start_date=2017-10-25&p_end_date=2017-11-01#130561

Created By: Marina Samuel

Start Date: 2017-10-24

End Date: Ongoing

Total Users: 1,161,007

Control Users: 578K

Variant Users: 580K

We worked with S&I to quickly launch a study in 57 Beta after seeing the 10pp drop in 56 release. 50% of Beta users were given the legacy (control) experience and the other 50% were given Activity Stream (variant). While the 56 study had multiple weeks of data, this one has about a 11 days [CONFIRM] of reported data as of today (Nov 5). We still feel like this is a useful test since we were able to see this issue in 56 during the first week as indicated in the second chart.

This is the 57 Beta data presented the same way: daily retention for all users stacked together so that their days zero align no matter their actual start date. We see the variant and control groups are retained at the same rate for both new and existing users.

This chart is using the same experiments data where we saw the problem in 56 Release.

7 of 21

56/57 Shield Study Summary

We saw a 10% drop in retention in the 56 release study using the experiments dataset.
We did not find that same result in the clients_daily dataset.
The initial drop in retention that we observed appears to be from missing data in the experiments dataset rather than users leaving Firefox. It’s still unclear at this time why data is missing.
We were not able to reproduce the retention drop in the 57 Beta study.

8 of 21

Data questions?

9 of 21

Contingency Plan

Stability: 50% of users in 57 beta have been using it for almost 2 weeks with no major issues reported. Crash rates between the two branches in Beta are statistically insignificant.

UX: We did a minor visual update to Tiles to make it a better fit with Photon. These have been uplifted and will be in Beta 14. Similar updates were made to about:home previously.

Onboarding: The onboarding experience is largely unaffected. We will need to restore the legacy option for data import since that now lives as a banner in Activity Stream.

10 of 21

Contingency Plan (cont)

QA: We have prepared a patch that disables Activity Stream, fixes tests, and restores the previous import flow. This has been validated by QA.

Legal: We confirmed with legal that there are no issues shipping the contingency.

Mobile: On Android and iOS we are planning on shipping Activity Stream as is with the option to disable Pocket stories to better align with our plans on desktop.

Support: We have a single page for our New Tab experience in SUMO. There could be some confusion for users visiting this page. A notice is being added to these pages that should help a bit.

11 of 21

Contingency Plan (cont)

Market Impact: Marketing believes it is best to not change their plans and are happy to handle this as a slow rollout over the next couple of months if need be. There is some individual concern about changing our story from the beta launch.

Snippets: No major technical issues with Snippets since they were designed to run on both experiences simultaneously. This does increase management costs and make optimization more complicated. This could make MoFo fundraising more challenging.

Pocket: Pocket stories in Firefox 57 is a kicking off point for a new business strategy for Pocket and delaying them could impact their goals for 2018. This is dependent on population size.

12 of 21

Contingency questions?

13 of 21

Shipping Options

Limited Rollout Fixed 1% rollout for the entire 57 cycle.	Gradual Rollout Start with 1% rollout and ramp up as we see positive results.	Full Rollout Pref ON by default. Immediately start two studies (0.5% existing and 0.5% new users preffed OFF) and draw down if we see any issues.
Pro: This would allow us to measure long term retention impact before rolling out to a larger audience.	Pro: Allows us to rollout to more people as we feel confident. Lots of flexibility (ramp fast or slow).	Pro: Maximum impact in contributing to Firefox “new”-ness. Clearest connection to planned marketing materials.
Con: No significant presence of Activity Stream in 57. Likely a more significant challenge for marketing.	Con: Mixed experience for users. May not match what they’re seeing in marketing materials or from other users/reviewers.	Con: A retention issue would impact all early 57 users. Drawing down significantly could make for similar marketing challenges as other options.

14 of 21

Next Steps

This Week

Communicate final rollout plan with experiment details.
Work with the Shield team to make sure we’ve done everything we can to prevent any future experiment retention issues.
Build dashboard to monitor ongoing release criteria.
Document release plan and schedule regular check-ins to review metrics (retention, engagement, perf, crashes, etc.) in release with subject matter experts.

15 of 21

Next Steps

Post Release

Review early data to see if there are any major retention issues. Adjust release parameters if there are any.
Regularly meet to review metrics and communicate status.
Monitor New Tab feature use and feedback so that we can make adjustments for 58.

16 of 21

Questions?

17 of 21

Appendix

18 of 21

Recent Quantitative Results

Overall Firefox 57 Beta usage remained consistent with Activity Stream on about:home.¹ Activity Stream gets higher engagement on about:home than about:newtab.²
Firefox Beta has the highest engagement rates that we’ve seen so far.³
We don’t yet know how this is going to translate to Release users.
Performance is critical to high engagement rates.⁴

19 of 21

Survey Summary

Overall satisfaction is consistent between Tiles & Activity Stream (4.5 stars).
36% of users read stories and articles in Activity Stream.
45% of users found stories and articles (sometimes - frequently) useful.
28% of users were concerned that Activity Stream displays sites they have previously visited (vs 23% of Tiles users).
~75% of both groups were extremely likely to use Firefox over the next few days.

20 of 21

User Research Summary

While users seem to generally understand how the different sections on New Tab work, there is still confusion among new users about their exact behavior.
Some users find Pocket stories to be a welcome distraction, while others were not interested in them.¹
Participants who visited Top Stories over the course of the study found them relevant, and thought that Top Stories were based on their browser history.²

21 of 21

User Research Summary (cont)

Users are not likely to interact with features like Highlights that are “below the fold”.³
Users cite many different reasons they want to customize New Tab.⁴
Many users are interested in customizing New Tab but not enough of them know that’s possible.⁵