CS 247 P4 - Milestone 3 (WOz Testing)

Team JJD: Jim Wang, James Liu, Daniel Greene

Overview: Wizard of Oz Testing

Three product ideas*:

​

• Musichat - Music-based communication
• Card Collage - Event-based communication
• Diary in Safe - Communication with self

​

Testing:

​

• Five users for Musichat
• Four users (and 30 email recipients) for Card Collage
• Six users for Diary in Safe

​

​

*Note: We chose to do three product ideas because we are not sure if all our ideas will fit within the realms of the class.

1. Musichat: Connect by music

Need: Share songs and communicate through music with friends.

​

Prototype: A chat service where friends communicate via sharing music (text is meant to embellish the music with commentary). The shared music is compiled into a playlist between those friends.

​

Quotes: “I want people to share songs with me because I don’t know which songs to listen to.”

“If I miss someone, I can listening to our shared playlist and contact them.”

​

Testing: Five unique users participated in each of the tasks.

​

Musichat: Testing

Task 1 (focusing on Role)

Question: How will the user respond after hearing a song played back to them in the chat conversation?

Rationale: Response is necessary to continue dialogue. We are gauging whether user is likely to respond right away, or whether they will put it aside.

Experiment: Once user enters chat history and plays the song in the conversation (we will play music via iPhone), we observe their response (if they reply with a music message, we help create that message and send it - see screenshot on right).

​

Task 2 (focusing on Implementation and Look & Feel)

Question: How well does the user engage with the UI? Are they confused about anything? Do they understand the purpose of the app?

Rationale: This task is necessary to identify glaring errors before coding.

Experiment: Have the user run through the app with no initial directions. Observe their interactions and their think-alouds. Interview them afterwards.

​

Note: We also tested other tasks such as how people like to discover music and record music, but for the sake of space we will not present them here.

Testing Task 1

Data and Feedback

Task 1

• 2 out of the 5 users replied back immediately with a song from their own collection. Another responded by “discover music.” The other two needed more time to think about the music.
• One user needed “space to focus on the music” and thus could not reply right away.

​

Task 2

• All users immediately understood that the app is about building music-centered relationships and communications with friends.
• All users said they will use the product. They liked the feature to make playlists with friends together. One user especially mentioned listening to a shared playlist will spark conversation.
• “I once saw on Spotify that my friend is listening to the same album as me and got really excited, but I cannot share that excitement with them. I will use this app because I can communicate about that shared music experience.”
• 2 out of 5 users found the button to add music confusing.
• “My playlist” was confusing to the first user. We subsequently changed it to “My favorites” and users were better able to understand it.
• One user wanted to share multiple playlists with the same friend.

​

Learning

Learning:

+ People do have a need to converse asynchronously with friends about music they are consuming.

+ People like discovering music through friends they trust more than receiving recommendations from music engines like Spotify.

+ People wish to add comments without first uploading music. The comments also don’t have to be arranged by time, as one may wish to comment on a music from awhile back.

+ People can understand the main needs and features from the UI interactions. Minor UI suggestions were helpful.

- People may not be keen to respond as readily to music as they are to regular texts. They have to be “in the right frame of musical mind.”

​

Design Changes

Based on the feedback and learning, we already implemented the following design changes and tested on more users:

• Changed “My Playlist” to “My favorites”
• Made a larger “Input music icon” and allowed testers to write their own messages.

Planned design revisions include:

• “Add comment” feature for each music post.
• “Queue music for later” feature if they are too busy to listen to the music right away
• Visualization of collective playlist beyond just “My fav” e.g. display musical connection with friends based on shared songs or number of playbacks.
• (Maybe) Develop for desktop where users have more “space” and “time” to listen to music.

​

Original

New design

2. Card Collage: Send a Collage Gift

Need: People need an easy way to collectively and remotely contribute to group messages, gifts, cards, and flyers.

​

Product: Card Collage - draw a card template, and then fill in the pieces with your friends to make a unique photo/video collage.

​

Tasks for testing:

• Are people willing to respond to an invitation to contribute? This is critical for the success of the product.
• What kinds of templates do people want to draw? This will determine how much flexibility we need to offer.

Prototypes

Prototype 1: Outreach email (Focusing on Role)

To test whether people would respond to an invitation, we emailed a mockup invitation to 30 friends and looked at the responses.

​

Prototype 2: Template sketching

(Focusing on Implementation)

To understand the space of possible use cases, we explained the product to four people and offered them a template to draw on.

Prototype 1: Feedback + Learning

• Out of 30 requests to contribute to Dan’s heart card, two people responded within six hours. Not bad - it’s an existence proof. And the results are delightful (see left).
• But the results may not generalize:
• Dan’s friends are Stanford students, who are atypical compared to the general public
• I acknowledged that I was looking for testing. A true test would be to present it as a real card to be sent to a loved one!
• So, less-than-ideal test, but useful to see that some people still respond.

Prototype 2: Feedback + Learning

• Two participants drew simple shapes, two wrote letters, and one made an elaborate design.
• One letter per square is an obvious and compelling idea for people!
• People want flexible numbers and arrangements of squares - five squares, squares stacked in different ways…
• Elaborate designs make it extra important to support accurate photos by overlaying the design on the camera image as you take the photo
• People want to reserve pieces so that they can work on them later!

3.Diary in Safe: Easy & Safe Mobile Diary

The need: People need an easy and safe way to record their diary entries but are too busy to write.

The product: A safe diary on mobile device that prompt daily for feeling and a message to self (text, audio, or video); users can go into the app and look at the entries and summaries later.

Representative tasks to test:

1. Get a push notification on phone and record an entry

- This is users’ most frequent (daily) interaction with the app, the diary part

2. Go into the diary and look at entries and summaries

- This is what users do within the app, and a representation of the “safe” part

Prototype and Testing

Prototype:

Two sets of paper prototypes in the form of flashcards, aiming at testing the two tasks

​

Testing and descriptive data:

Wizard of Oz testing by showing flashcards conditioned on users’ actions, followed by interviews, 6 tests in total:

- 6 of 6 correctly understood the interaction by looking at the flash cards

- 4 of 6 said they will use the product regularly

​

Feedback and Learning

Feedback:

+ Push notification is a key driver for using the app

+ The low friction, i.e. only need to select feeling, and messages are optional

+ The summary curve is a great way to visualize entries

- Some people seldom go into the diary to look up certain entries

- Recording messages might be hard without facilitating questions

- Filters on negative feelings could seldom get used

​

Design Change Directions:

• Can link with calendar to remind user of events
• Should consider allowing users to grant access to others

​

Summary

• Prototyping was hugely helpful in clarifying our designs and revealing users’ perspectives
• Prototyping was most helpful when it focused on answering specific design questions, as opposed to testing the overall flow of a UI
• What will a user do when they receive a song?
• What kinds of card designs do people make?
• What will a user do when they get a push message?
• Prototyping provided concrete ideas to make our projects better
• Queuing music to hear later
• Overlaying card designs on the camera to take better photos
• Allowing diary users to share select entries
• We liked both Musichat and Card Collage and will discuss with teaching staff about technical feasibility and course requirements