Can we improve privacy without breaking the web?
Privacy Settings
Breakage Study
August 2017
Brought to you by ...
Jacqueline Savory
Interaction Designer
Luke Crouch
PrivSec Engineer
Peter Dolanjski
Product Manager
Brought to you by ...
Ryan Harter
Browser Measurement
Ilana Segall
Data Science
Years of awesome
Privacy engineers
Existing Claims
Tracking Protection breaks websites
Broken websites make users leave Firefox
Some existing prefs could protect users with minimal breakage
Opt-in page
On-boarding
Report:
“page problem”
“page works”
Breakage type
Notes
Thank you!
Disable Study
The numbers
https://sql.telemetry.mozilla.org/dashboard/shield-study-improve-privacy-settings
19,000+ users
9 branches
https://github.com/mozilla/shield-study-privacy
2,100+ users in each branch
https://sql.telemetry.mozilla.org/queries/17837#73402
Up to 8,500 active users/day
https://sql.telemetry.mozilla.org/queries/23123#60018
About those claims ...
Tracking Protection breaks websites
Broken websites make users leave Firefox
Some existing prefs could protect users with minimal breakage
Does Tracking Protection
break websites?
Avg. problems reported per user looks lower for trackingProtection ...
https://sql.telemetry.mozilla.org/queries/23721#61701
Avg. problems reported per user looks lower for trackingProtection ...
WTF?
https://sql.telemetry.mozilla.org/queries/23721#61701
Some control users’ problems ...
“Something* on the page is slowing down the loading speed significantly.”
“not responsive”, “slow, freezing”, “Took longer than usual for page to load”, “Connection appears slower than usual”, “Pages are scrolling slowly”, “very slow to load”, “long wait for anything to occur”, “the fire fox not always responding”, “page is very slow to load”, “tremendous lag , page loads very slowly”, “page was laggy and didn't respond”, “Sending mail in Gmail is very slow since installation of this study”, “really slow to load”, “video doesn't load fast”, ...
Tracking Protection
may actually fix websites by blocking tracking elements that break/slow them down!
Do broken websites make users leave Firefox?
Do broken websites make users leave�this study?
Some common site breakages ...
resistFingerprinting causes Facebook problems
firstPartyIsolation causes YouTube problems
https://sql.telemetry.mozilla.org/queries/18276#61772
What % of users leave the study?
After reporting breakage on certain popular sites
Don’t break SUMO: 72% of users left
Don’t break Google sites: 6 out of top 25
Don’t break email: Yahoo Mail, Gmail, Outlook Live in top 20
https://sql.telemetry.mozilla.org/queries/27989#73748
What % of users leave the study?
After reporting breakage on longer tail sites
Don’t break dev sites? Atlassian, GitHub show up
Don’t break porn sites? xvideos show up
https://sql.telemetry.mozilla.org/queries/27989#73748
Breaking workflow sites
(search, accounts, email,
support, development)
makes users leave this study
Do certain kinds of breakage make users leave this study?
“other” is most common breakage* ...
* new claim: we’re not good at predicting kinds of breakage?
https://sql.telemetry.mozilla.org/queries/19634#50162
Some common breakages
firstPartyIsolation causes login failures
resistFingerprinting causes flash problems
https://sql.telemetry.mozilla.org/queries/19634#61483
What % of users with certain breakage disable the study?
94% of users reporting screen breakage disable study
84% of users reporting flash breakage disable study
82% of users reporting login-failure breakage disable study
64% of users reporting payment breakage disable study
https://sql.telemetry.mozilla.org/queries/20097#51471
What % of users with certain breakage disable the study?
screen and flash are only in resistFingerprinting
payment is in control & 3DP cookies branches
login-failure in control, 3DP cookies, first-party isolation, & referer branches
https://sql.telemetry.mozilla.org/queries/20097#51471
Across all branches,
breaking logins and payments
makes users leave
this study
Are there existing prefs that could protect users with minimal breakage?
14% of control users report breakage
18% of firstPartyIsolationOpenerAccess users:� the max recorded in the study
https://sql.telemetry.mozilla.org/queries/23644#61485
6 settings are within margin of error of control
.21 avg. problems per control user
.25 thirdPartyCookiesOnlyFromVisited
.19 trackingProtection
https://sql.telemetry.mozilla.org/queries/23721#61701
4 settings are within margin of error of control
5.1% of control users disable study
8.5% of firstPartyIsolation users
4.7% of originOnlyToThirdParties users
https://sql.telemetry.mozilla.org/queries/19633#50159
5 settings are within margin of error of control
How can we compare “overall breakage”?
“Composite Breakage Score”
An index of web breakage
% of users who report breakage
Average breakage reported by each user
% of users who disable the protection (presumably because of breakage)
*
*
“Composite Breakage Scores”
“Composite Breakage Scores”
WTF?
.24 noThirdPartyCookies
.24 sessionOnlyThirdPartyCookies
.27 thirdPartyCookiesOnlyFromVisited
WTF?
https://sql.telemetry.mozilla.org/queries/23721#61701
Some thirdPartyCookiesOnlyFromVisited users’ problems ...
“The message tells me that my cookies are blocked even though my settings are to accept cookies.”
“Got this message ... Cookies are blocked. … your browser doesn’t allow cookies. ...change your browser settings.”, “Cannot access on onlyine bill pay because it thinks cookies are blocked. I checked an my options say to allow cookies. what is going on?”
Users don’t understand
Accept third-party cookies: From Visited
?
Most promising prefs
Based on “Composite Breakage Score”
originOnlyReferer�ToThirdParties
trackingProtection
sessionOnly�ThirdPartyCookies
User values
originOnlyRefererToThirdParties
🙂 Reduces detail sent to trackers
🙂 Very few login failures
🙂 Very little mail breakage
Ecosystem values
originOnlyRefererToThirdParties
User values
trackingProtection
🙂 Blocks known trackers completely
🙂 Speed boost
🙂 Very little mail breakage
😢 Triggers ad-blocker-blocker walls
Ecosystem values
trackingProtection
😢 Blocks ads
User values
sessionOnlyThirdPartyCookies
🙂 Limits duration of tracking
🙂 Very little mail breakage
😢 Some login and “unexpected signout” failures
Ecosystem values
sessionOnlyThirdPartyCookies
🙂 Does not blocks ads
Key take-aways
Tracking Protection doesn’t seem to “break” websites as much as we feared
Breaking workflow sites makes users disable the study
Yes!
Some existing prefs could protect users with minimal breakage
More: post-study user surveys
Next ...
Q3-Q4 2017
Tracking Protection
UI for “Always”
launched
in Quantum
https://bugzilla.mozilla.org/show_bug.cgi?id=1387681
Private Browsing
https://bugzilla.mozilla.org/show_bug.cgi?id=587523
Can trackers just use IP addresses
to track Firefox users?
IP Address Study
https://bugzilla.mozilla.org/show_bug.cgi?id=1423748
Next ...
2018