ABCDEFGHIJKLMNOPQRSTUVW
1
ProtocolMozilla Public
2
ParticipantP1 (pilot)P4 (pilot)P6 P8P9Task Average Scores
3
TaskQuotes & ObservationsGrade PDRQuotes & ObservationsGrade PDRQuotes & ObservationsGrade PDRQuotes & ObservationsGrade PDRQuotes & ObservationsGrade PDRGradePDR
4
Onboarding: Welcome Screen*no video, just audio* (screen recorder issue)*no video, just audio* (screen recorder issue)7.0 OS8.0 OS
5
Without touching your device, please describe what you would normally do next after opening this app for the first time. REMOVED 3/19[USABILITY] Participant skips quickly through onboarding screen first time through, describes log in screen. "I just enter the email address in to load it up properly"N/A[USABILITY] Participant skips quickly through onboarding screen first time through but realizes he's done it.N/A[USABILITY] Participant skips quickly through onboarding screen and taps Get Started "I would tap Get Started and I've already done it"N/AREMOVED 3/19N/AREMOVED 3/19N/A
6
Without touching your device, please describe what you see on your screen."just enter your email and the firefox logo"N/AN/AReads contents of login screen, then reopens app to go back to splash screen.N/AN/A(Participant skips quickly through blue onboarding/landing screen and taps Get Started)
"I would tap Get Started and I've already done it" "I saw a sign that said 'Get Started' so I tapped it and it took me to this [the sign in] screen"		N/AN/A(Reads all the text on the splash screen aloud) "I see a loch-ness monster, a 'Get Started' button and 'Learn More' and just information about what FIrefox Lockbox is"N/AN/A"it's just says FIrefox Lockbox, there's a little loch-ness monster... take your passwords everywhere, 'Get Started'. It looks like you have to have an account with some saved logins."N/AN/A
7
Based on what you see on the screen titled "Firefox Lockbox", what do you expect Lockbox to do for you?[MENTAL MODEL] Participant (accurately) expects Lockbox to fill in the passwords for all the items in the list wihtout typing (not convinced she's means autofill though)
"Once I've put in passwords for the websites, once I've logged them in here, I expect it to be able to fill them in automatically for me without having to type them in every time."		1N/A[MENTAL MODEL] "I think I missed the splash screen" "I would click the 'get started' button. (reads splash screen content) "ok, i understand"
"I expect that it's somewhere that i can keep my passwords & its probably accessible here maybe on my other devices." (continues reading) "i guess i would probably click that learn more button to see if there's anything else it would do for me"		0.5N/A[MENTAL MODEL] (referring to sign in screen, sounds unsure)
"If I were to enter my email address that I use to open a Firefox account, I think it would enable me to see all my devices that are synchronised. I think so, I'm not sure"
(does not mention password management) 		0.5N/A[MENTAL MODEL] "I'm guessing it's going to allow me to save my login username and password. So maybe it's a standalone application that I can use on multiple web browsers, not just FIrefox, which will allow me to save my login information."0.5N/A[MENTAL MODEL] "I think that its going to save all of my passwords in Firefox. "0.5N/A0.6
8
What, if anything, would you do on the screen titled “Firefox Lockbox”?[USABILITY] Participant says 'skip for now' aloud (could be biometric or autofill permissions), lands on Entry List - empty state... and then pauses video & fixes sync in her FxA account & sets up Lockbox autofillN/AN/A[USABILITY] "I would probably try to figure out what the other benefits are other than keep it in a notepad file or notebook or something" Taps 'learn more.' Scans FAQ "this is really good info to have once I get started, but for me now i want to see what the benefits are of keeping my passwords in Lockbox" (more reading, returns to splash screen & taps get started.N/AN/A[USABILITY] (skipped blue onboarding/landing screen, answering about the sign in screen)
"I am going to enter the email address that I used to open the Firefox account" and taps Continue		N/AN/A[USABILITY] "I would hit 'Get Started' to start."N/AN/A[USABILITY] "I would just click 'Get Started'N/AN/A
9
Onboarding: FxA Sign In
10
You should now be on a screen titled “Enter your email.” Without touching your device, what email address would you use to log in? [do they use Firefox email?][MENTAL MODEL] Uses FxA email address, understands connection b/w FxA & Lockbox.
"I used my gmail address, the one that's linked with firefox" 		1N/A[MENTAL MODEL] Uses FxA email address. "i would use my gmail account that i previously used on some of the other documentation"1N/A[MENTAL MODEL] "I would use the email address that I used to open the Firefox account" 1N/A[MENTAL MODEL] "I would use [email], the email associated with my FIrefox account."1N/A[MENTAL MODEL] "Just use my gmail account that I use the most."1N/A1
11
Use the email address you use to log into your Firefox account to log in to the app. When you get to the screen that says “You’re all set!” at the top, stop and go to the next task.[USABILITY] (completed task successfully but in a very round about way, due to skipping or misundertanding protocol tasks, but then reports that signing in with Lockbox was "easy enough to log in")0.55[USABILITY] BUG: "That's what I meant" (referrring to email used for FxA) then something happens? "i know that's not the app it's my phone" gets "invalid token" error tapping continue button on email confirmation screen15[USABILITY] (on login screen, while typing password) "i don't like the fact that you can see [her password]" "this is so slow, maybe it's my phone" (shows spinner + "Working" loader message for 40 seconds before going to "You're all set!." at 30 seconds, she says her tablet dinged to tell her there was a new sign in. Lockbox did not display the email confirmation screen. Lockbox did not display autofill or fingerprint screens, per @sheinen, this is because user was on Android 7.0)15[USABILITY] Moves quickly through onboarding/setup without issues.14[USABILITY] Forgets password to log into Firefox account, makes multiple attempts at entry, pauses test to reset and then tries again a few more times. 0.530.84.4
12
What do they do on the email confirmation screen?[USABILITY] (not sure - no video or audio clues)N/AN/A[USABILITY] (not sure - no video or audio clues)N/AN/A[USABILITY] BUG? SCREEN DID NOT DISPLAY?N/A[USABILITY] Checks laptop to confirm sign-in.N/AN/A[USABILITY] BUG? SCREEN DID NOT DISPLAY This screen did not show in the video - after entering her password, it skipped straight to Fingerprint screenN/AN/A
13
What do they do on the Fingerprint screen? [USABILITY] (says 'skip for now' aloud, doesn't enable)N/AN/A[USABILITY] (not sure - no video or audio clues)N/AN/A[USABILITY] SCREEN DID NOT DISPLAY per @sheinen, this is because user was on Android 7.0?N/A[USABILITY] (Reading) enables fingerprint "uh, ok"N/AN/A[USABILITY] Quickly taps "Skip" without reading aloud. Does not enable.N/AN/A
14
What do they do on the Autofill screen?[USABILITY] (says 'skip for now' aloud, doesn't enable)N/AN/A[USABILITY] "I missed this - it's easy to miss this type of splash screen as an average Joe" "it looks like one of those artibtrary 'click here to allow this app to access your contacts' type screens"N/AN/A[USABILITY] SCREEN DID NOT DISPLAY per @sheinen, this is because user was on Android 7.0N/AN/A[USABILITY] (Reads aloud) "Am I supposed to do this?" "So I guess I have to do this." Enables autofill.N/AN/A[USABILITY] Quickly taps "Skip" without reading aloud. Does not enable.N/AN/A
15
Do they enable Fingerprint?N?N/AYN
16
Do they enable Autofill?N NN/AYN
17
Onboarding: Lockbox Onboarding Confirmation
18
(Biometrics) Before you tap anything, please quickly answer a few questions. During Lockbox setup, did you see a screen inviting you to enable Fingerprint for Lockbox? If so, how did you respond and why? REMOVE 3/19[USABILITY/MENTAL MODEL] "yes I did and I skipped it because I didn't know whether I needed to do it or not"N/AN/A[USABILITY/MENTAL MODEL] "i almost remember a fingerprint icon but i don't rememeber anything specifically asking for that and my current device doesn't have fingerprint so that might be why i didn't see it"N/AN/A[USABILITY/MENTAL MODEL] "I didn't—and I'm assuming it's because—I don't know why. Because I have the option on the phone, however I know I had it deactivated and I don't remember seeing a popup screen inviting me to enable Fingerprint" "If one had popped up for Fingerprint, unless it's absolutely necessary I probably would not have activated the Fingerprint because I don't like it—I prefer using PINs or pattern, I don't know why"N/AN/AREMOVED 3/19N/AN/AREMOVED 3/19N/AN/A
19
(Autofill) During Lockbox setup, did you see a screen inviting you to enable Autofill for lockbox? If so, how did you respond and why? REMOVE 3/19[USABILITY/MENTAL MODEL] Intially skipped but then comes back & enables it later. "i thought that was what [autofill on Lockbox] was for so that'd make it easier and make the Lockbox useful"N/AN/A[USABILITY/MENTAL MODEL] DIDN'T ENABLE AUTOFILL DURING SETUP: "i did see that screen but for some reason my brain ignored it a little bit - i don't thnk i quite realized there was another step there. i don't know what i expect to see but that screen seemed almost like an arbitrary screen that could easily be skipped"N/AN/A[USABILITY/MENTAL MODEL] "I didn;t see anything inviting me to autofill either. If I had seen it I would probably have enabled autofill."N/AN/AREMOVED 3/19N/AN/AREMOVED 3/19N/AN/A
20
(All Set) Bringing your attention back to the current screen, without touching your device, what do you expect Lockbox to do for you? After responding, tap Finish and go to the next task.[MENTAL MODEL] Partially correct: assumes passwords already in her browser will be in Lockbox, and that she will access them through Lockbox... but doesn't mention going to an app or website and having Lockbox autofill.
"I imagine that I would be able to save different passwords that are already in there—they'd save them into my Lockbox & then everytime that I'd click on a website that was already saved, the passsword would automatically be in there."		0.5N/A[MENTAL MODEL] Partially correct: assumes her FIrefox passwords will be in Lockbox, and that she will access them through Lockbox... but doesn't mention going to an app or website and having Lockbox autofill.
(reads bullet) "I understand that this is just going to allow me to use - to get passwords from Firefox onto my phone specifically, not on any other device i have around—it's good that I clarified that because I wasn't sure before." (reads bullet) "Makes me feel pretty secure about putting my passwords on here I think I guess because obviously as a consumer, putting my passwords anywhere online, even if its a reputable company like Mozilla, still makes me feel a little uneasy, but just seeing the fact that you guys put this here makes me feel a little more at ease." (reads bullet) "now i'm seeing the fingerprint item - it makes it a little more secure feeling for me because i guess they need my fingerprint as well"
"from this screen—what i'm expect Lockbox to do for me—is that i can pull out my phone, i can look at all my passwords maybe that I have have for FIrefox, maybe that I have for other devices that I can manage in the system and that it will be secure for me to put them in there for me and secure so that no one else can get into this app." (taps finish)		0.5N/A[MENTAL MODEL] Correctly assumes Lockbox will allow her to access saved logins from all her devices that use FIrefox, and autofill them elsewhere.
(reads the 'You're all set' screen content aloud, then says referring to Fingerprint text— "I deactived that for my phone, I may have to reactivate it" "Is probably to allow me those things that I was just questioned on—to choose to enable fingerprint unlocking option, and it would allow me to autofill as well, being as I would be able to access logins that are saved to FIrefox from my phone. And maybe also for other devices, since it would be synchronized between the devices. 		1N/A[MENTAL MODEL] Partially correct: assumes FIrefox passwords will be in Lockbox... but doesn't mention going to an app or website and having Lockbox autofill. (Reads text) "I expect to be able to access my logins that I've saved to my Firefox browser on my phone, I guess."0.5N/A[MENTAL MODEL] Partially correct: assumes FIrefox passwords will be in Lockbox... but doesn't mention going to an app or website and having Lockbox autofill. (reads bullets) "It's basically just saving my access logins on Firefox with my phone." 0.5N/A0.6
21
Initial Impressions: Credential List
22
Without touching your device, what is your first impression of what you see?[MENTAL MODEL] (initially sees No Entries Found due to user error setting up test) "Very basic - gives information into how to view entries if I had them" (thinks she needs to sync, then she pauses video, syncs & credentials appear) N/AN/A[MENTAL MODEL]"so here's all my stuff, oh my goodness" "i knew this was going to happen because i signed a consent form so i'm not worried what's happening next, but just seeing some of the things in here... i guess that i didn't realise were saved by Firefox, just makes me at first a little 'whoa' that this is all already being saved on here I guess, just as an average consumer" (scrolling list) "my capitalone passwords are in here, that's a little terrifying to me. so knowing that this is already being saved is a little bit of a shock, but at the same time, if this is something i was going to use to easily pull any of this information it seems very convenient that it's already being saved. And now thinking back to the security and encryption settings, the fingerprint i can put in place for this if i had it enabled on my phone, makes me feel a little bit better. definitely very convenient." "it's a little overwhelming just because there's so much here that i actually didn't realize was being saved onto this device or through my account or something."N/AN/A[MENTAL MODEL] (participant realizes now that Firefox has been saving her logins) "oh my goodness. OH MY GOODNESS." (scrolling) "There are SO MANY of my accounts here." (reads names of accounts & scrolls) "this is very interesting." "my first impression is 'What the heck has been happening in the background!? Where'd they get all these—of my stuff from?'" "what the heck, they've been keeping my stuff" "I'm shocked as to what was happening all along."N/AN/A[MENTAL MODEL] (chuckling) "Oh, ok. That's interesting." (reads protocol question while checkling) "I guess I'm a bit shocked because I'm surprised that it's showing me all my usernames for all the different sites. I'm a bit shocked that it knew all this information and it was able to figure it out."N/AN/A[MENTAL MODEL] (initially sees No Entries Found—guessing this is beause of all the password changes?) "No Entries found—looks like I need to sign in & sync to have it show up." "Uh oh, maybe I haven't synced it yet. Oh my." Taps around hamburger menu: Account, Settings, FAQ—looking for a way to sync her account.
Pauses video & comes back, can see list (2 entries—firefox account & mozilla account).
N/AN/A
23
Go ahead and explore this screen. Where did this list of apps and websites come from?[USABILITY/MENTAL MODEL] Matter-of-factly: "that's all my passwords that I imagine they got from all the accounts so that I'm linked in wth everything...Lockbox because I'm synced in it's got all my details from all the different websites I use, so they're there"1N/A[USABILITY/MENTAL MODEL] "i guess that's what i'm trying to figure out. i'm assuming it's coming maybe from my Google account, or maybe not, it's very interesting, some of these are things I've done at work so I'm assuming it's anything that's tied to any firefox sessions that i've had potentially."
Isn't sure initially, and is surprised to see stuff he did pre-FxA here (doesn't realise he's saving stuff in the browser & that creating a FxA makes them available in additional places): "there's alot of things in here that i didn't expect to see that i maybe used once and never again. so i guess that would be my guess. it's either coming from my Google account or coming from sessions I've had on Firefox browser from different websites—that's kind of what i'm thinking too. I know some of these things I've done I've primarily used Mozilla Firefox so I'm assuming it's kind of captured that for me, which I find interesting only because I have only created my Firefox account recently and I have used some of these websites at work. So yea i assume that's where it's coming from."		1N/A[USABILITY/MENTAL MODEL] (correct—understands what the list is)
"Maybe it's Firefox—all things I've been keeping on Firefox, they've been saved." "More than likely they came from my using/opening these applications through Firefox" (opens a detail page and then another [see step 20 for think aloud]) "it seems basically that once I have used Firefox to open an account somewhere, it stores my user name and password—especially maybe in cases where I may have opted to autosave the username and password."		1N/A[USABILITY/MENTAL MODEL] (partially correct)
"This list came from Firefox browser on my mobile phone" [interesting, because he's also a Firefox desktop user]
Notices search, filter, hamburger menu.		0.5N/A[USABILITY/MENTAL MODEL] "I think it came from my Firefox account. That has saved from my Firefox account I guess.1N/A0.9
24
Show us how you go about finding one of your logins to use for a website or app.[USABILITY] "There's all the entries I recently used" (sorts by alphabetically) "I can do alphabetically and change them. So let's see if I want to see if there's one for recent ones, I scroll down & see" (scrolls, names some websites)15[USABILITY] (scrolling list) "It's in alphabetical order, which is nice"
skips ahead:
(taps on a row, Lockbox locks itself, he enters his PIN) "instead of my fingerprint I guess it's just using that [his PIN]." "ok that's interesting. I expected that when I clicked on it, it would open up my passwords for that website, but it looks like it just tried to log me back in, send me back to a splash screen or something?" (tries again, it works) "to reveal my password, I would click on the little eyeball here"		14[USABILITY] (scrolls and then taps on a detail, but does not sort, search or notice how they're organized. understands 2/3 usecase paths)
"I would go and then tap on one of these options" "if i had forgotten I could copy the username & copy or look at the password" "spooky and interesting at the same time"
(expresses confusion about protocol because she thinks 'login' means 'password')		0.55[USABILITY] (scrolls a bit & taps on a row—does not search, sort, or comment on how they're organized) 0.55[USABILITY] "I think I just click on one of these [rows] and it shows me my information." (scrolls a bit & taps on a row—does not search, sort, or comment on how they're organized) 0.550.74.8
25
Uses sort? (1=Y, 0=N)Yes1??No0No0No00.25
26
uses search?????No0No0No00
27
Uses scroll?Yes1Yes1Yes1Yes1Yes11
28
Without touching your device, what would you expect if you tapped one of the items on the list? REMOVED 3/19[USABILITY/MENTAL MODEL] "I would guess now that it would come up with the details about the website login and that it would show the password, obviously hidden, maybe the the dots for the password and the last login possibly."N/AN/A[USABILITY/MENTAL MODEL] (already did it earlier) "I guess i would expect for it bring up just my password but i guess it's giving me a lot more information than just that. It's good that it was hidden by default."N/AN/A[USABILITY/MENTAL MODEL] (understands how to view details)
"it would show me more options, which is what i thought would happen"		N/AN/AREMOVED 3/19N/AN/AREMOVED 3/19N/AN/A
29
Initial Impressions: Credential Details
30
If you haven’t already, tap one of the items on the list. What are your impressions of what you see? Did it match your expectations? Why or why not? REMOVED 3/19[USABILITY/MENTAL MODEL] Meets expectations. "It's got the web address, it's got the username, and it's got the password in there and I can have a look at it if I want to. So it's pretty much what I expected. The only thing I thought it might have is the date I last logged in. Everything else is as expected and easy to read & access."N/AN/A[USABILITY/MENTAL MODEL] (already did it earlier) expected just to see a list "i like that you can copy & paste some of these items and it looks like you can just click on the url icon to go there—it looks a little more interactive than i would expect" (reads) "learn how to edit this entry—that's interesting as well."N/AN/A[USABILITY/MENTAL MODEL]"oh my goodness. so this—i don't even remember when I did this. Obviously I did it on Firefox at some point. This gives me the web address, my username and my password, and I can even see the password, which I'm not gonna press" "and I can copy and I don't know what this is—to go there?—the top arrow. oh my goodness" "I wasn't expecting the option of copying username & whatnot—it exceeded my expectations"N/AN/AREMOVED 3/19N/AN/AREMOVED 3/19N/AN/A
31
Without touching your device, if you wanted to view your password for an entry, what would you do?[USABILITY] "If I press on the eye it would show the actual password, and I imagine that if I pressed on the pieces of paper it would copy the password and I'd be able to put it in the website myself. "1N/A[USABILITY]"it's so intuitive i just kind of did it... pulling this up with people around...i'd feel good that it was hidden & the eyeball is pretty universal to use as unhiding it."1N/A[USABILITY] (correct)
"i would simply click the eye (clicks it) and it would show me the password"		1N/A[USABILITY] "I would press the eye button"1N/A[USABILITY] "I think I would click on the eye"1N/A1
32
Without touching your device, can you imagine a scenario or two where you might want to view your password for an entry? If so, explain the scenario. If not, explain why not.[MENTAL MODEL] "I guess if I were trying to log in somewhere from a different computer that didn't have my details - like maybe a public computer - I'd be able to see my password there instead of constantly trying all different ones & getting it wrong."1N/A[MENTAL MODEL] "um...why would i use this device?" describes general use cases for using a password manager( "paying bills online") and hints that maybe he'd use it to view passwords at work (presumably on a device other than his phone, but doesn't explicitly say that)1N/A[MENTAL MODEL] "i may want to view just in case i have forgotten it - i may want to write it down" "or if we need to sign in from another device—i guess it helps in cases when one may be moving from one device to another unexpectedly"1N/A[MENTAL MODEL] "When I really have no idea what my password is and I want to, just out of curiosity, figure out what it was." "Maybe I want to sign in to the website on my laptop with a different browser—maybe Safari. If I want to do that I can just look at my password by clicking the eye icon and entering it manually into Safari."1N/A[MENTAL MODEL] "If for some reason [my password] didn't come up on FIrefox or if I wanted to use it for a different device or something then I may want to view it"1N/A1
33
If you needed to change your password for this entry, how do you think you would do that? (Don't actually change your password)[MENTAL MODEL] (notices 'learn how to edit this entry link' but doesn't tap) Assumes non-existant functionality: "I would imagine that I could just click on where it says 'password' and delete and put another one in, but I am not entirely sure"0N/A[MENTAL MODEL] Correct "I'm assuming I would go to "learn how to enter this entry" "otherwise i would go to the website its saved on. honestly if i didn't see the blue link i wouldn't know i could do it from here"1N/A[MENTAL MODEL] (partially correct)
"i don't know quite frankly. i think would click on the button that says 'learn how to edit this entry' I believe it would allow me to edit my password and I believe Lockbox would automatically save everything right there" (does not actually click it)		0.5N/A[MENTAL MODEL] "I would click the password field—the dots—and just change it to whatever I want."0N/A[MENTAL MODEL] (correct) "I think I will click the two pages, it looks like an edit" (taps "copy" icon) "well maybe not. Maybe the thing on the bottom that says 'learn how to edit this entry'." (does not actually click it)1N/A0.5
34
Return to the list. If you wanted to add a new username and password to this list, how do you think you would do that? (Don't actually add a new username and password to the list)[MENTAL MODEL] (scrolls to look for an add to list button). "I scroll down, there's nothing there" Looks in Account "not entirely sure how you could add another one"0N/A[MENTAL MODEL] (scrolls down to look for add to list button) "i'm looking for a plus, but there's not." (scrolls back up) "i'd expect almost something at the top of the list to look like a little plus...or if there was like a floating icon with a plus on it that would probably be what i'd expect, but beacuse that's not there i guess i'm going to click on the menu button." (views hamburger) "hm, maybe the only way to add ones of these is to create an account somewhere on (by "on" he seems to mean "while using") Firefox and then it would it it to this list"1N/A[MENTAL MODEL] (misunderstanding question) " select the link that says 'learn how to edit this entry' and I could change it"0N/A[MENTAL MODEL] "I have no idea" Scrolls, opens hamburger menu Settings & Account, tries search, opens hamburger menu again, eventually gives up and repeats "I really don't know how to do that actually."0N/A[MENTAL MODEL] (misunderstanding question) "I think I would click 'learn how to edit this entry' is how I would edit it."0N/A0.2
35
Locking the App
36
If you’re not already on the list, please return there. Let’s say you want to lock and close the Lockbox app. How you would lock it?[USABILITY] Taps on hamburger, taps 'lock now'. Once locked, explains that she can unlock by tapping button. "That was easy. Easy to find, easy to know what to do"15[USABILITY]"i will go to menu here—Lock Now" "I probably wouldn't have known it was something I could just lock, so I probably would have just shut the app down without having looked at the menu here."0.54[USABILITY] (correct)
(taps hamburger menu) "i would go here and then click 'lock.' i wonder what locking it does, though. i suppose i would just close it afterwards." 		15[USABILITY] Opens hamburger "I would hit the 'Lock Now' button over here."15[USABILITY] Returns to list, opens hamburger and closes it without seeing 'Lock Now' opens it again, then sees it "There;s this thing that says 'Lock Now' I click on that"150.94.8
37
If you haven’t already, tap the menu icon in the top left corner, then tap Lock Now. Is this how you expected to lock the app?[USABILITY]"very quick simple easy to do, straightforward, where I expected it to be"N/AN/A[USABILITY]"i would almost expect that if I closed the app I have to log in too, to unlock it. i wouldn't expect to click a button that's hidden in a menu—maybe it's something floating over the list already"N/AN/A[USABILITY]
(locks it successfully)		N/AN/A[USABILITY] (locks it successfully) "Yes it is" N/AN/A[USABILITY] "Yes that was super easy to find"N/AN/A
38
Retrieving Lockbox Credentials
39
For the next task, you will be asked to use this app to help you sign in to an external account on your mobile device, either an app or a website. Please unlock the Lockbox app. Without tapping on any entries, look at the list and decide on an account you would like to access. N/AN/AN/A(scrolls, chooses Humble Bundle)N/AN/A[USABILITY] Taps unlock on splash screen & gets the device PIN entry screen. is worried/confused at first "uh oh, I don't know what it is" (then enters her device PIN) (surprise—PIN interaction is not in line with her expectations) "AH! it's my regular phone lock. oh my goodness i almost freaked out, because i did not set up a lock code" "I wasn't expecting it to use my lock code from my device—but that's very functional, very good because i don't need to create a new PIN of any sort. very good, i'm impressed"N/AN/A[USABILITY] Successfully uses Fingerprint to unlock app. N/AN/A[USABILITY] DIDN'T ENABLE FINGERPRINT DURING SETUP Successfully uses device PIN to unlock app. N/AN/A
40
Without touching your device, how do you imagine that you could use the Lockbox app to help you to sign into the external app or website you chose?[MENTAL MODEL] Assumes non-existant functionality & misses functionality: Expects Lockbox to deep link into other apps and log in automatically when she chooses an item from the list. Does not expect to go to the app/website first.
"I would imagine that I just click on it, and where it says where the website, on the (credential details) page, I expect I would be able to click on that & it would take me to the login page automatically"
0.5N/A[MENTAL MODEL] Assumes some functionality. Does not expect to go to the app/website first.
"I'm assuming I would just click on Humble Bundle and it would show me my password. knowing now that I unhide the password by clicking on the eyeball, I would do that. And then I can potentially just type it in from knowing what's there on like another device maybe. Or flip back and forth on my phone to check the password and flip back to the Humble Bundle website and type my password in there. It looks like you guys made it easier... so I can just click the copy button and click to open the link so, that, again, is beyond what i expected for ease of use."
0.5N/A[MENTAL MODEL] (partially correct—expects nonexistant functionality, but seems like a form of copy/paste)
"i would click on [app entry], and i would click on arrow link to open website. i'm wondering if it would automatically sign me in since i'm opening the website from Lockbox, but i'm not sure"		0.5N/A[MENTAL MODEL] "That's a good question. I guess you have to click into the website [in Lockbox?]—into that specific entry you want, and from there maybe there's a button you can press that says "log in to this website." 0.5N/A[MENTAL MODEL] "I think I would just click on one of these (referring to list). I would check on the username & password just right there. 0.5N/A0.5
41
Autofill by going to app/website first? No0No0No0No0No00
42
Autofill by opening app/website in Lockbox?No0No0Yes1Yes1No00.4
43
View & then type?No0Yes1No0No0Yes10.4
44
Copy & paste?Yes-ish0.5Yes1Yes1No0No00.5
45
Retrieving Lockbox Credentials Autofill
46
Follow these steps:
Step 1. Close Lockbox.
Step 2. Open the app you chose from the earlier question.
Step 3. In the app you chose from the earlier question, tap the username field and stop.

What do you see? What do you think will happen if you tap the box that appeared?		[USABILITY/MENTAL MODEL]DIDN'T ENABLE AUTOFILL DURING SETUP: (opens Duolingo, taps into UN/email field, Lockbox autofill affordance doesn't appear) "I didn't see anything - I'm not sure why"0N/A[USABILITY/MENTAL MODEL] DIDN'T ENABLE AUTOFILL DURING SETUP: closes Lockbox, opens Humble Bundle website using link on Lockbox detail page, signs out of website—UN & PW is already autofilled. doesn't have any apps on his list, so he tries another website. When he taps on a UN field, he sees a floating box with two lines: his email address & a CTA: Manage. "it looks like both of those are pulling from my ANdroid phone, maybe not from Firefox—I really can't tell."
feels like this is browser functionality, not Lockbox."If I click the box that appears, I'm expecting it will just in my username there" 		0N/A[USABILITY/MENTAL MODEL]
BUG: DIDN'T GET AUTOFILL SCREEN DURING SETUP, doesn't understand that Lockbox can be used from within other apps to autofill
(misunderstands protocol question & taps on username in Lockbox, does not open external app to see autofill) "how the heck am I supposed to close - i can't close it without using [copying?] it [the username?]" (taps username row & is surprised that it copies without her touching the copy icon)		0N/A[USABILITY/MENTAL MODEL] BUG - autofill overlay appearance is very delayed. Closes Lockbox by hot swapping & swiping up to force quit. Opens website in browser. UN & Password are already present in the fields. Keyboard opens when he taps in UN field, but no Lockbox overlay appears. Autofill overlay does not appear intially "I didn't see any box"
Decides to try another website. Returns to browser. Focus is in UN field and Lockbox "search your entries" overlay briefly appears, but participant appears not to notice and dismisses it. Types in the new URL. 		0N/A[USABILITY/MENTAL MODEL] DIDN'T ENABLE AUTOFILL DURING SETUP Opens Firefox browser and goes to support.mozilla.org (UN & PW are already there, she deletes them). Autofill overlay does not appear (since it's not enabled). Tries to sign out & can't figure it out, gives up.
"I think that if I was to tap the username field and stop, I would see a box, and the box would put the password into the password box" 		0N/A0
47
Now tap the box. What just happened?[USABILITY/MENTAL MODEL] DIDN'T ENABLE AUTOFILL DURING SETUP: autofill affordance doesn't appear - tries again several times, returns to Lockbox, scrolls to find password manually) "didn't seem to work on that app"03[USABILITY/MENTAL MODEL] DIDN'T ENABLE AUTOFILL DURING SETUP: Concludes that autofill is coming from Android & not FIrefox. "I'm assuming what should be happening is if I click into one of those either on FIrefox browser or my Android browser that something should be popping up to manage it or to drop it in. If what's supposed to happen is that it just shows me my login & I click it and it just drops it in for me, that's great. it's really hard to distinguish wether it's my phone doing it or whether it's the browser doing it."03[USABILITY/MENTAL MODEL] BUG: DIDN'T GET AUTOFILL SCREEN DURING SETUP
(has not opened an external app to see autofill)
"there is no box" [expects to access external websites through Lockbox, expects a sort of autofill—but not the designed happy path—does not mention accessing an external app or website first] but also mentions tapping on website URL in Lockbox again to open app and then "i ahve the option of putting in my usernames and then looking at my password and typing it in and it would probably surprise me and autofill both the username & apssword."		05[USABILITY/MENTAL MODEL] BUG -Lockbock Autofill overlay does not appear
Opens new website. Keyboard opens when he taps in UN field, but no Lockbox overlay appears. Lockbox Autofill overlay does not appear. Confused: "Maybe I have to open up an app?" Taps in password field, system autofill overlay appears. In the background, he's hearing notifications. He opens system notificaitons center & sees he's received a series 3-4 of scary push notifications saying "leak detected." Participant is confused by this, thinks the protocol is referring to the debug messages and explores those. "I have not idea what just happened."

(Per @dreams, the debug messages are unrelated to autofill & a function of the debug build) 		01[USABILITY/MENTAL MODEL] DIDN'T ENABLE AUTOFILL DURING SETUP Deletes username & password, taps in username field, a [non-Lockbox?] autofill overlay appears, she taps it, and her password is filled automatically. "When I click it, it puts the password in for me"0503.4
48
Overall impressions
49
Reflecting on your use of Lockbox today, did it work the way that you expected it to? Is there anything you would change?[USABILITY/MENTAL MODEL] Says it works well, then blames self for bug.
"It seemed to work very well when I was looking through to websites that were already on there, the only issue came when I was trying to log in because i've not got many apps on my phone that i've got logins for—it didn't seem to work on that Duolingo one so I think that was more likely down to my phone rather than the app—the rest of it works really smoothly and it was easy to set up and navigate"		N/AN/A[USABILITY/MENTAL MODEL] "Honestly, I don't think so, it actually did more than I expected it to do—I honestly just expected it to be a list of usernames, what it was used for and passwords. But it definitely made the process more easy to use. Again, the list is a little overwhelming when you first see it but I geusss that's just the nature of what you're trying to do here. "
"And I think the interface to lock it—it might be handy to make it a little more obvious if that's something that you wanted. But once you do it once, it's easy to know how to lock it down. But it does seem like if you close it without locking it down you could very easily get back into it. So that's one minor thing. "
"There should be another way to add passwords in here, than just creating them within the Firefox browser—I would make that more apparent, because I don't really see a way to do it other than that. If that's the way it's supposed to work, than I wouldn't change a thing."		N/AN/A[USABILITY/MENTAL MODEL]
"there are some things i wasn't expecting - for example, the list when i opened the Lockbox, but it made sense afterwards that i'm able to see all of the list because of the fact that I opened and signed in to an account from Firefox and probably opted in to saving my signin credentials." "the ability that if the link is open from the Lockbox, that it would not [require signing in] because it's already synced."		N/AN/A[USABILITY/MENTAL MODEL] "Well, everything except the actual filling out of the login information worked as I expected it to." "At this point, maybe just make sure that it automatically does fill out my information—in this case it didn't. Other than that, there's nothing else I would change."N/AN/A[USABILITY/MENTAL MODEL] "it was a little confusing with all the usernames & trying to get into Firefox. I love being able to have them all in one place and then having them fill in the passwords for me once I put in my username. It definitely worked the way I expected it to. I guess Lockbox kind of keeps them in one palce, so if I had to access it, then I could." N/AN/A
50
How does this experience using Lockbox compare to your current method for access and retrieval of usernames and passwords?[USABILITY/MENTAL MODEL] Sounds like Firefox may not be her main browser (or she doesn't save logins in it) she seems to think she needs to take steps to get credentials into lockbox. "It should be easier once I've got all my accounts set up - I've got them all on different accounts, different websites, different email addresses, so I think once that's all set up it would be alot easier to keep them all in one place securely rather than just keep forgetting them or having to reset them all."N/AN/A[USABILITY/MENTAL MODEL] "I don't really use a lot of these password manager apps because I feel like the access is going to get into the wrong hands—that sort of thing. But it's way easier than using it like in a book like I have in the past because when I'm at home or at work, now I know I can access these passwords from anywhere because I always carry my phone with me. I definitely feel like this is way easier to manage because there are passwords in here that I think I've forgotten even exist...WHich is really nice, because going back to those places and now knowing 'oh i already have an account here, here's the password' that's something that I would never have even thought of. ANd I can tell you that maybe 90% of these passwords are not in the notebook that I've created."N/AN/A[USABILITY/MENTAL MODEL] "it's kind of helpful in one breath and frightening in another breath. and i say helpful because, oftentimes, if i'm with my laptop but not with my desk/notepads that i may have jotted down information for an account—and i won't always have the notebooks—Firefox saves these passwords and whatever else that would be needed. And having the passwords readily available on the page helps"
"using Lockbox compared to my current method—writing down things in a notebook—is definitely a big jump. definitely helpful." "oftentimes when i don't remember [my password], I have to create a new password... Would be better not having to recreate passwords all the time" 		N/AN/A[USABILITY/MENTAL MODEL] "It's definitely more complicated. You have to install an app, create an account, log into an account and then the app has to work in the background—if it doesn't, the error will come up & then you can't use it. So it's quite complicated compared to just using Firefox with the saved passwords." N/AN/A[USABILITY/MENTAL MODEL] "If I saved it on a website [in the browser] then it does pull up already, but I guess Lockbox is handy because if I was on a different device, or for some reason it wasn't pulling up as being saved, I could go back to Lockbox and see what my password is" "It's a much better situation than what I'm doing now, where I think I've saved this one but I never know until I'm there."N/AN/A
51
Do you think that Lockbox could help you to improve the security of your passwords? Why or why not?[MENTAL MODEL] Understands that lockbox can support & empower her to improve her password hygiene: "yes as long as it's save & secure & can't be hacked or anything - i think it would improve the security of my passwords so that I don't keep using the same ones all the time maybe. They're all in one place, hopefully safe"N/AN/A[MENTAL MODEL] Equates password security with PIN & Fingerprint, but doesn't think of the passwords themselves: "I think it depends on how well you have your phone locked down. I have the PIN right now, if I had fingerprint I think it would feel a little more secure. And it feel like it would be more secure if that lock button was a little more obvious...maybe not on the main screen, where it's easy to hit & accidentally lock things down, but i feel like it could definitely improve security of my passwords just because you know anyone could just like open my notebook that i have sitting around my house and not everyone can get into my phone—I don't normally let it out of my sight. And knowing that you guys encrypt the data definitely makes me feel more secure about it, but I still feel that leery kind of feeling just having it on—this is obviously stored somewhere digitally. And I understand it's encrypted and I can only access it on my side with my PIN and how I securely lock down my phone. So I guess it feels a little more secure to me than having them out in the open in a notebook somewhere.N/AN/A[MENTAL MODEL] (describes the security of having synced password across devices that are accessible by Lockbox, but not maintaining unique passwords)
"Yes and no. Having this kind of system with Lockbox, my passwords would go with me wherever I am, regardless of the device I am on—I suppose it would help." "but phones can be lost or stolen and one wouldn't want to have a quick click on the app or website." (doesn't think that would happen because she expects Lockbox would automatically lock after not being used for an hour)
(reatcting to final 'remove Lockbox' instruction: "OH you want me to remove it? I don't want to remove it unless it's not a real app. I think I'l keep it for a bit."		N/AN/A[MENTAL MODEL] "i'm not quite sure at this point. I think the only thing I could maybe benefit from is viewing my password over here. That's pretty cool." "I don't think it would improve my security because it pretty much does the same thing as a password manager with the exception that it lists all the usernames and passwords, and I can view the passwords."N/AN/A[MENTAL MODEL] (Understands that Lockbox can support & empower her to improve her password hygiene) "Yes definitely. Especially because I'll use similar ones—having them in a palce where I can use more obscure ones but not having to remember them all—that would improve the security of my passwords."N/AN/A
52
Final thoughts?"No comments except for it was a bit tricky to set up at the beginning, I'm not sure why my duolingo one wouldn't work even when I tried putting in the username it didn't work so I think that was more likely down to my phone rather than the app. It looks really good and easy to use""N/AN/A"If there is a way that this should be accessed on the device, i'd say i only use the default browser on here—I did go into some apps and see if anything happened, but I did not see it work the way I'm guessing it should be working. SO i guess if it is trying to pull in information or update information, I'm not seeing it but, if it is being pulled in specifically being pulled in from the Lockbox app, I may be expecting to see some sort of icon there that tells me that that's where the information is being pulled from—a little Lockbox icon maybe—and maybe that's what's supposed to pop up, I'm really not sure. I couldn't seem to replicate that. But that's how i'd expect it to happen if it is being stored on my Firefox Account. I'd also expect to probably be able to access that on my computer when I'm on Firefox as well and I'm not really seeing that. It looks like the little key icon which could've been tehre before, could not have been there before. But I guess that's what I expect to happen. If anything other than that is happening, I'm definitely not expecting it to happen, so.""If it's not synchronized across devices—all my devices would need to have the Lockbox installed, I'm assuming so. If not, it would be nice to be able to just open a browser and be able to get on Lockbox fror the web rather than having to download an app."none"I feel Lockbox was really helpful in saving passwords for all of my site—I only had a few, but I would be interested in getting in and adding more."
53
General Notes
54
no video! had to guess on some actions based on narrativeno video! had to guess on some actions based on narrative
55
Participant Average Grade0.61538461544.50.730769230840.615384615450.53846153853.750.64.50.52222222224.35Group Task Average Grade
56
Participant Average Difficuly RatingGroup Average Difficuly Rating
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100