Remote Usability Test: Shake-A-Leg Miami
CIM 622 UX Research Remote Test Report
Test Date: April 4- 20th, 2018
Report Date: April 25th, 2018
Moderators: Morgan McKie, Juniette Fiore, Angie Del Llano, Nicole Barba,
Haosheng Li, Yaozhi Zhang, William Sun
For all inquiries, please contact Professor Barbara Millet at bmillet@miami.edu
1
2
Content
Executive Summary
3
Executive Summary
4
Shake-A-Leg Miami is a registered nonprofit organization that offers people with disabilities an opportunity to share challenges and successes through the organization’s programs and activities.
Our group of seven researchers first conducted a moderated (lab-based) usability test on the organization's website. That test was shortly followed by a remote usability test which included 49 participants. In the remote test, there were 19 problems and 5 positive attributes identified. Of those 19 problems, 14 of them were the same on both tests and 5 were found exclusively through the remote test. The issues found solely through the remote test included issues with font style, lack of search bar, inconsistent navigation, location of navigation bar, and placement of information.
We also found that participants had a harder time navigating through the Shake-A-Leg Miami website in the remote test and were more candid with their frustration during the study.
The positive aspects of the site included the quality of images, informative content, donation button, social media links, staff contact information. The next steps would include research methods to explore appropriate information architecture for target users.
Introduction
5
Introduction: Product Description
6
The shakealegmiami.org website offers information about Shake-A-Leg Miami (SALM), a registered non-profit organization where people with disabilities can share challenges and successes through the programs and activities offered by the organization.
Currently, the Shake-A-Leg website uses a template created by Vizergy, which offers templates and custom websites targeted towards the hospitality business. This site was last updated March 15, 2018.
Introduction: Current Site Map
7
Introduction: Test Objectives
8
The purpose of this study was to identify usability issues by collecting data and feedback from each test session. The results collected helped inform the redesign of the site navigation, ensuring each task is achieved with ease.
Introduction: Client’s Needs
9
During our first meeting with Mr. Harry Horgan, the founder of Shake-A-Leg, he expressed that he wanted the site to:
Methods
10
Methods: Test Facility (Remote)
11
This is a remote usability test with a total of 49 participants who took the test via an external link, sent to them via email. The study was conducted using UserZoom, a UX Research Platform, in a location designated by each participant
Methods: Tools and Equipment
12
Methods: Procedures
13
Methods: Tasks
14
Task 1: Identify what Shake-A-Leg is.
Scenario: You want to know what Shake-A-Leg is. Find information about it and write a sentence telling what Shake-A-Leg is.
Task 2: Become a volunteer through the Mentoring Program.
Scenario: Shake-A-Leg is primarily run by volunteers. You are interested in becoming a mentor. Find the form to allow you to become a mentor. Do not fill it in.
Task 3: Find the first session available for Water Sports Camp
Scenario: One of Shake-A-Leg’s main objectives is providing programs that interest their target audience. You want to join a water sport. When is the first opportunity to participate in water sports?
Task 4: Make a $25 donation.
Scenario: Shake-A-Leg accepts donations to run their programs. Make a $25 donation to the Wellness Center. Do not submit the donation.
Task 5: Find the news.
Scenario: You want to read updates about Shake-A-Leg. Find the latest news.
Task 6: Find information about Impossible Dream.
Scenario: Shake-A-Leg has a sailing boat that is for disabled use. Find information about it on Shake-A-Leg’s site.
Methods: Usability Metrics
15
Time on Task: Record time on task allowed us to measure the effectiveness of the Shake-a-Leg website.
Task Completion
First Click: Records participant’s first mouse click on the website’s interface.
Usability Issues
Usability issues are anything that affect how effectively and efficiently a product is used and user satisfaction with the product.
Severity Scores
Severity Scores a rating from 0 (lowest) to 8 (highest) of how impactful a usability issue is the functionality of a product.
Content Analysis
Content Analysis is a technique for systematically describing spoken comment during the Usability Test. It provides a quantitative (numerical) description of positive and negative voices.
System Usability Scale (SUS)
The System Usability Scale (SUS) is a 10-item questionnaire used for measuring the usability of a product. It uses a scale from a 0 to 100 scale. A SUS score below 68 would be considered below average.
Methods: Post-Study Scores
16
The System Usability Scale (SUS) was used to evaluate the site’s usability.
The SUS consists of 10 questions, with a five-point scale, ranging from Strongly Agree to Strongly Disagree.
The SUS was originally created by John Brooke in 1986, which allows you to evaluate a wide variety of products and services, including hardware, software, mobile devices, websites and applications.
Question 1: I think that I would like to use this website frequently.
Strongly
Disagree
Strongly
Agree
1 2 3 4 5
Source: Brooke 1996
Methods: Participant Profiles
17
Sample Size: 49
Methods: Demographic Summary
18
Sample Size: 49
Age
Hispanic or Latino
18 - 21
years old
23 - 29
years old
Over 29
White
Asian / Pacific Islander
Some college credit or college graduate
Master’s Degree
Race
Highest Level of
School Completed
Black or African American
21 - 23
years old
Yes
No
2%
Some high school or high school graduate
4%
Other
Methods: Demographic Summary
19
Sample Size: 49
2 - 4 hours
Over 4
hours
Student
Employed
Hours Spent on the Internet Per Day
Employment Status
Performed Usability Test Before
Made a Donation to an Organization
Yes
No
Familiar with Disability Services
No
2%
Unemployed
Retired
Less than 1 hour
Yes
No
Yes
Performance & Satisfaction
20
Severity Scale
21
This is the breakdown of the severity scores for each issue.
There are a total of 19 issues spread across six tasks, each ranked by how much effect they’ll have on the Shake-A-Leg website. This means that there were:
These scores were generated by finding the average of Jeff Rubin’s severity and frequency scales.
Low Severity
High Severity
7
6
5
4
3
2
8
Unique Improvement Opportunities
22
This is the breakdown of the unique Improvement opportunities found in the remote test.
We found a total of 5 unique improvement opportunities out of 19 total improvement opportunities. The improvement opportunities unique to the remote test are:
Unique Improvement Opportunities
23
This is the continued breakdown of the unique Improvement opportunities found in the remote test.
We found a total of 5 unique improvement opportunities out of 19 total improvement opportunities. The improvement opportunities unique to the remote test are:
Completion Rates
24
Task 1: Identify what Shake-A-Leg is.
Task 2: Become a Volunteer through the Mentoring Program.
Task 3: Find the first session available for Water Sports.
Task 4: Make a $25 donation to the Wellness Center.
Task 5: Find the latest Shake-A-Leg News update.
Task 6: Find information on Shake-A-Legs sister site; Impossible Dream.
This table shows how many participants successfully completed each task.
Task completion rates indicate that the information is not easily visible on the site and needs to be restructured.
<1%
Completion Rate Comparison
Remote vs Lab-Based
25
<1%
Task 1: All participants in the lab-based test were able to complete this task
Task 2: Participants in the remote test had a harder time finding the volunteer form
Task 3: Participants in both tests found the task (almost equally) difficult to complete
Task 4: Participants in both tests found the task (almost equally) difficult to complete
Task 5: Lab based testing had betters results for this task
Task 6: In both tests, only 1 participant was able to complete the task successfully
Time on Task Comparison
26
Task 1: Identify what Shake-A-Leg is.
Task 2: Become a Volunteer through the Mentoring Program.
Task 3: Find the first session available for Water Sports.
Task 4: Make a $25 donation to the Wellness Center.
Task 5: Find the latest Shake-A-Leg News update.
Task 6: Find information on Shake-A-Legs sister site, Impossible Dream.
This graph shows a comparison between the average amount of time taken for each task between Remote Testing (sample set of 49) and Lab Testing (sample set of 7) .
Similarly to the the Lab testing Task 3 is the task that took the longest and is one of the two tasks that had the lowest completion rate.
Although Task 6 completion was on average 8 seconds quicker, but was successfully completed by a single participant. This is comparable to Lab testing where the average was 16 seconds quicker but had the same out come of only having one participant successfully complete it.
Satisfaction Scores Comparison
27
Task 1: Identify what Shake-A-Leg is.
Task 2: Become a Volunteer through the Mentoring Program.
Task 3: Find the first session available for Water Sports.
Task 4: Make a $25 donation to the Wellness Center.
Task 5: Find the latest Shake-A-Leg News update.
Task 6: Find information on Shake-A-Legs sister site, Impossible Dream.
The Single Ease Question (SEQ) is a 7-point rating scale completed after a participant completes a task. It assesses how difficult the participant found the task.
Tasks 1 and 4 still had the highest average SEQ scores, which shows that participants in the Remote testing found these two tasks very easy to complete but half of the participants in the Remote Testing group where not able to complete Task 1.
Also only 31% were able to complete Task 4.
Task 3 changed from 5.1 to 3.6 in remote testing. Making task 3 and 6 the lowest average SEQ scores, showing that participants felt this task was the most difficult to complete.
SUS Scores
28
The System Usability Scale (SUS) is a 10-item questionnaire used for measuring the usability of a product.
From a 0 to 100 scale, a SUS score below 68 would be considered below average.
Shake-A-Leg received a SUS score of 46 in the remote test versus the 42 received in the lab-based test. This is in the direction of low perceived usability.
Average SUS Score (lab-based) N=7 | 42.14 |
Average SUS Score (remote) N=49 | 45.92 |
Issues Found: Remote Test
29
The are total of 19 issues found in remote test. 5 unique issues found in remote test
9 issues related to Content, 3 issues related to Visual Design, 8 issues related to Navigation, and 3 issues related to Layout.
Content Analysis
30
Task | Positive Comments Percentage | Explanation | Negative Comments Percentage | Explanation |
Task 1 | 60% | Positive comments included appreciation of the site’s photos and statements about the information being easy to find. | 40% | Negative comments included dissatisfaction with the location of the navigation bar, complaints about the blue color of the font, the description of the organization being vague. |
Task 2 | 20% | Positive comments included appreciation informative content and information being easy to find. | 80% | Negative comments included a inability to easily search for needed information, poor information architecture, poor location of link to form, and very small font size. |
Task 3 | 10% | Positive comment included appreciation of hover function on main navigation bar. | 90% | Negative comments included frustration with blank pages, inconsistent sub navigation placement and redundant navigation bars. |
Content Analysis is a technique for systematically describing spoken comments. It provides a quantitative (numerical) description of positive and negative voices.
Overall, 14% of all comments were positive while 72% of all comments were negative.
Content Analysis
31
Task | Positive Comments Percentage | Explanation | Negative Comments Percentage | Explanation |
Task 4 | 30% | Positive comment included easy of use. | 70% | Negative comments included poor location of the Donate Button and very long form. |
Task 5 | 10% | Positive comment included an appreciation of informative content. | 90% | Negative comments included poor selection of font color, no current information, inconsistency of content layout and poor information architecture, redundancy a navigation bars, |
Task 6 | 0% | There were no positive comments. | 100% | Negative comments included small font size, broken links, and links that redirect to a google search page. |
Keepers & Opportunities
32
Keepers
33
34
Staff Contact Information
Providing staff member names, roles, and emails allow users to easily contact the right person for specific questions.
Voices:
“It gives you the names of the staff and the chance to email them if you have any questions. That’s really nice. It would be nice if they had a phone number though.”
Keeper found in: Remote Test
Keeper Location: http://www.shakealegmiami.org/leadership-team.htm
35
Images/Videos
Images and videos of participants are currently visible on every page (in the headers or embedded in the
content on each page) which provides a lot of information to the users. This gives site visitors a preview of the activities offered by Shake-A-Leg and a sense of community. It also gives potential donors a taste of what Shake-A-Leg is about if they have yet to visit the facilities or look deeper into the website content.
Voices:
"The pictures show what the organization is about."
"Pictures are helpful, overall interesting and high quality photos."
"The website is attractive thanks to photos."
"I liked the pictures. People are participating and actively doing things."
Keeper found in: Remote and Lab-Based Test
Keeper Location: http://www.shakealegmiami.org
36
Informative Content
There is a lot of detailed information displayed throughout the site. The information provides people with knowledge of what Shake-A-Leg is all about, it’s purpose, and the many activities offered.
Voices:
“In general, information is placed well and pretty straightforward to find what I need to.”
"The site is very informative."
Keeper found in: Remote and Lab-Based Test
Keeper Location: http://www.shakealegmiami.org
37
Donation Button
A "Donate Now" button is placed in the top right of the main page. The location is obvious, making it easily accessible for users. However, participants instinctively first searched for it in the main navigation menu.
Voices:
"It is straightforward and obvious on the page."
"This task is easier than others because it's very prevalent on the home page."
“The donate button is clear and easy.”
Keeper found in: Remote and Lab-Based Test
Keeper Location: http://www.shakealegmiami.org
38
Social Media Links
Social media is the most effective way to engage and connect with users as well as drive more traffic to the website. Users looked to the social media links to learn about updates and news.
Voices:
“Social Media seemed like the closest option to find updated news.”
Keeper found in: Lab-Based Test
Keeper Location: http://www.shakealegmiami.org
Improvement Opportunities
39
40
Remote Test
Lab-Based Test
Overlap
5
4
14
Improvement Opportunities:
Remote Test vs Lab-Based Test
In total, 23 improvement opportunities were found between the remote usability test and lab-based usability test.
While there were 19 improvement opportunities found in the remote test, 14 of those overlapped with those found in the lab-based test. Only 5 of improvement opportunities were considered unique to the remote test. In comparison, the lab-based test had 4 unique improvement opportunities.
Task 1: Identify what Shake-A-Leg is
Scenario: You want to know what Shake-A-Leg is. Find information about it and write a sentence telling us what Shake-A-Leg is.
41
Task 1 Clicks & Paths
42
Data was gathered through observation of each participant’s video and tracking their click paths. Data available in spreadsheet.
First Clicks
For Task 1, the most common first click was the About page on the main navigation bar. This first click was made by 28% of the participants (14 out of 49) and was counted as a ‘Pass.’
The second most common clicks were Who We Are at 22% (11 out of 49) and Shake-A-Leg’s Story at 22% (11 out of 49). Who We Are was counted as a ‘Pass,’ but Shake-A-Leg’s Story was counted as a fail.
The two least common first clicks were Testimonials & Photos at 2% (1 out of 49) and Water Sports Camp at 2% (1 out of 49).
Click Paths
The most common click path to reach the correct response was clicking the About page on the main navigation bar, which was followed by 28% of participants (14 out of 49).
Pass: Task completed successfully
Fail: Task not completed successfully
43
Improvement Opportunities:
Font Style Issues
Some participants noted that the application form is difficult to find because the font is too small and dissimilar to the rest of the site. Others pages on site have a larger font and are bolded. Also, some pages had font colors in blue, gray, or black.
* This issue appeared in all tasks.
Recommendation: Make the font uniform across the site. Use the font Helvetica Neue (regular style) in font size 12 for content text. Use Helvetica Neue Bold for headings. Make the text color for the body black since it is easy on the eyes.
Voices:
“The blue is kinda hard on my eyes..”
“The hue is weird.. I don’t like the blue font, black is fine.”
Problem Category: Visual Design
Problem found in: Remote Test
Problem Location: http://www.shakealegmiami.org/
Successful
Completion: 25/49
5
Low Severity
High Severity
44
Improvement Opportunities:
Removing Redundancy
Participants found the About Us page in different ways. Some went to About while others went to Who We Are, which is located in the dropdown menu under About on the main navigation. The information on both pages are similar but slightly different.
Recommendation: To facilitate interaction, create a single About Us page.
Problem Category: Content
Problem found in:
Remote & Lab-Based Test
Problem Location:
http://www.shakealegmiami.org/who-we-are.htm
http://www.shakealegmiami.org/about-us.htm
3
Low Severity
High Severity
Successful Completion: 25/49
45
Improvement Opportunities:
Message Clarity
Although the participants completed the task easily, many of them felt that the description itself could use some work. Both by shortening it and rearranging the information. Some participants felt it was vague.
Recommendation: Rearrange the information enclosed in red (Image 1 and Image 2) so that Shake-A-Leg’s main focus (helping people with disabilities) is in the first paragraph.
Voices:
“This description is kind of vague. I know they do something with disabilities, but what exactly? I guess they do outdoor activities with disabled people.’
2
Low Severity
High Severity
Problem Category: Content
Problem found in:
Remote & Lab-Based Test
Problem Location:
Successful Completion: 25/49
Image 1
Image 2
46
Comparison: Remote Test vs Lab-based Test
Improvement opportunities found in the remote test were similar to the ones found in the lab-based test. The only new improvement opportunity was the inconsistent font style throughout the website.
In both tests, participants noticed that the About Us page was vague in its description. In both tests, participants found information about the organization in two main ways: About Us and Who We Are.
Improvement Opportunities Unique to Remote Test: 1
Remote Test | Severity Score | Lab-based Test | Severity Score |
Removing Redundancy | 3 | Removing Redundancy | 3 |
Message Clarity | 2 | Message Clarity | 2 |
Font Style Issues* | 5 |
|
|
*Improvement opportunities found that were unique to the remote test.
Task 2: Become a volunteer through
the Mentoring Program
Scenario: Shake-A-Leg is primarily run by volunteers. You are interested in becoming a mentor. Find the form to allow you to become a mentor. Do not fill it in.
47
Task 2 Clicks & Paths
48
Data was gathered through observation of each participant’s video and tracking their click paths. Data available in spreadsheet.
First Clicks
For Task 2, the most common first click was “Education and Mentoring”.This first click was made by 37% of the participants (18 out of 49) and was counted as a ‘Pass.’
The second and the third most common clicks were “Supporters and Sponsors” at 18% (9 out of 49) and “Giving Opportunities” at 10% (5 out of 49). “Supporters and Sponsors” and Shake-A-Leg’s Story were counted as a fail .
The least common clicks results in a 7 way tie between “Corporate Engagement”, “Staff”, “Adult Volunteer”, “About”,”Events”,”Shake a Leg Wellness Center” and “Navigation”. Of these 1st clicks, all led to failure except “Adult Volunteers” and “Events” led to a fail.
Click Paths
The most common click path to reach the correct response was: “Miami Mentor Program”->form.jotform.com/73494373238161. This path was chosen 43% (21/49) of the time with a success rate of 26% (13/49) .
Pass: Task completed successfully
Fail: Task not completed successfully
49
Voices:
“I don’t think they should put Employment Opportunities under Programs & Activities. Maybe it should have a career tab or a join us tab.”
“Since this is primarily run by volunteers, I feel that something that important should be near the top of the page, not hidden.”
Improvement Opportunities:
Navigation Confusion
Not only was it difficult for users to find the right page, they were often confused by other page titles, such as Supporters & Sponsors. Additionally, some participants felt that the placement of Employment Opportunities under Programs & Activities was confusing and that it should be placed elsewhere.
Recommendation: Have all volunteer opportunities listed in one place by creating a Get Involved tab. Get Involved includes the following:
Create a Work With Us page to list available employment opportunities. See Redesign.
Problem Category: Navigation
Problem found in:
Remote & Lab-Based Test
Problem Location:
5
Low Severity
High Severity
Successful
Completion: 28/49
50
Improvement Opportunities:
Form Location
It was difficult for some participants to find the application form. It is not hyperlinked when it is first mention so participants have to scroll to the bottom to find it.
Recommendation: In step 1 of the FAQ, make the words “Fill out the application” a hyperlink to the application form.
Voices:
“I see that the form is all the way at the bottom of the page. If you’re trying to encourage people to take a look at this website and volunteer, they’re going to miss that important piece of information because it’s hidden all the way at the bottom of the page.”
Problem Category: Layout
Problem found in:
Remote & Lab-Based Test
Problem Location:
4
Low Severity
High Severity
Successful
Completion: 28/49