5 of 20

Think Aloud Procedure

N = 3

Relevance:

Participants were business owners, homeowners, people that work for the city

Zoom

Cell phones

Desktops/Laptops

Figma

Think aloud. Listening as participants perform tasks, prompt for more thinking vocally if necessary

Participants

Tools

Method

Gain insight into what steps users take as they go through the tasks, why they take them, and other related thoughts they had

Goals

6 of 20

Usability Testing Procedure

N = 10

Relevance:

Participants were business owners, homeowners, people that work for the city

Figma

Cell phones

Desktops/laptops

Randomized the order
Tested 2 tasks on both systems
Survey after each

Participants

Tools

Method

Time on task

Error rates

User satisfaction

Goals

7 of 20

User Satisfaction Survey

Participants: N = 10

Method: After each usability test, participants filled out survey on current and new system

Tools: Google Forms

Goals: Improve user satisfaction through improved recoverability, error prevention, and visibility of system status

9 of 20

Think aloud sentiment analysis

Current system mean = 0.367 and standard deviation = 0.153
New system mean = 0.800 with a standard deviation = 0.100
T-test = -4.10
P-value = 0.015
Statistical significance!

Current vs New:

10 of 20

Think aloud results

Current system:

New system:

“I don't know what would make sense there.” - participant 1

“Yeah, way easier than what I just did” - participant 1

“That's very straightforward. Oh, yeah, I like this” - participant 9

“But it’s not letting me click it, so I don’t really know.” - participant 7

11 of 20

Usability Test Results

Summary Statistics - Task 1

Current System

Mean: 83.29

Standard Deviation: 56.55

Redesign System

Mean: 47.69

Standard Deviation: 45.19

Summary Statistics - Task 2

Current System

Mean: 624.41

Standard Deviation: 94.01

Redesign System

Mean: 216.42

Standard Deviation: 58.44

T-Tests

Task 1: t(5) = 5.84, p-value = 0.0021

Task 2: t(5) = 9.03, p-value = 0.0000136

Statistical significance!

Not Transcript Note: if asked why we used unequal variance t-test for task 2, say, “the variance within the task 2 participant groups was unequal when comparing variance by variance ratio, which showed to be over 2.00 for task 2 groups. Being over 2 meant the variances were unequal so required an unequal variance t-test.�

Transcript below:

* Usability Testing, Chart 1 shows task 1 across systems. The redesign overall shows it takes lesser time to find the permit than in the current system.

* Chart 2 shows task 2 across systems. The redesign overall shows it takes far lesser time to fill-in the permit than in the current system.

T-test results:

Task 1 (Finding the Permit):� A paired t-test comparing the current and redesigned systems for Task 1 (within-subjects) showed a statistically significant difference in time.

T-test with 5 df, and t-value of 5.84, t(5) = 5.84, is a large value indicating a substantial difference in task time.
With a p = 0.0021 confirms this effect is statistically significant, suggesting the redesign meaningfully impacted performance.

Task 2 (Filling Out the Permit):� An unequal variance t-test comparing the current and redesigned systems for Task 2 (within-subjects) revealed a statistically significant difference in completion time.

T-test with 5 df, and t-value of 9.03, t(5) = 9.03, indicates a very large difference between systems.
With a p-value less than 0.05, p = 0.0000136, confirms this result is highly significant and that the redesign strongly affected task performance.

Overall, this suggests that our new design drastically improved the Task 1 and Task 2 performance, reducing the time to navigate for a permit and fill-out the permit.

* While comparing means across task 1, with within-group participants, a paired t-test against task 1 current system vs. task 1 redesign showed there was a statistically significant difference in time to find the permit.

· Based on the task 1 test statistic, with df = 5, the t-test of 5.84 is very large, relative to the variation in the data, meaning the difference in time for task 1 between the old and new systems is substantial. And paired with p-value of 0.0021, this strongly suggests the new system significantly changed task performance time.

* While comparing means across task 2, with within-group participants, an unequal variance t-test against task 2 current system vs. task 2 redesign showed there was a statistically significant difference in the time it took to fill-out the permit.

· Based on the task 2 test statistic, with df = 5, the t-test of 9.03 is also very high, relative to the variability in the sample, meaning the difference in time for task 2 between the old and new systems is large. When paired with the p-value = 0.0000136, which is below 0.05, means the result is statistically significant.

12 of 20

User Survey Results

Did you feel “stuck” at any point while using the site (e.g., trouble with navigation, unable to find your way through the site)?
Was it clear how to get back on track after an issue?
Were instructions and field labels clear enough to avoid mistakes?
Did the site keep you informed about what was happening (e.g., loading, saving, submitting)?
Were status messages (e.g., success/failure) easy to understand?
Did loading indicators or progress bars help you understand what was happening?
Overall, how satisfied were you with your experience using the mobile site?

13 of 20

Inferences

03.

14 of 20

How the Redesign Improved Usability

Question 1: Improved 2.8 → 3.5

Question 2: Improved 2.3 → 4

Question 3: Improved 2.3 → 4.1

Recoverability

Error Prevention

Question 4: Improved 1.9 → 4.6

Question 5: Improved 1.9 → 4.5

Question 6: Improved 1.5 → 4.4

Question 7: Improved 1.9 → 4.7

Visibility of System Status

Satisfaction

All four usability goals were supported and none were compromised by the changes in our redesign. First, Recoverability. In the current system, users often felt “stuck” and unsure of what to do next. The average recoverability score was just 2.55 out of 5. To improve this, we simplified the navigation flow and added clearer step-by-step indicators. These changes helped users feel more in control, and the average score jumped to 3.75. So, recoverability was strongly supported by our redesign.

Next, Error Prevention. One of the biggest issues in the original system was confusion in the fees section and missing required fields. On average, users made 14.5 errors, and the error prevention score was just 2.30. We addressed this by improving field labels, adding real-time validation, and restructuring the fees section for clarity. After these changes, the score rose to 4.10—a huge improvement—so this goal was also very well supported.

Third, we looked at Visibility of System Status. This was the lowest-performing category in the original system, with an average score of 1.77. Users often didn’t know if their form was saving or submitting correctly. In the redesign, we added loading indicators, progress bars, and clearer success or error messages. That change led to a big improvement, with the new average score at 4.5. Visibility of system status was very strongly supported.

And finally, User Satisfaction. In the current system, satisfaction scored just 1.9. We redesigned the interface to be cleaner, mobile-friendly, and more intuitive. As a result, satisfaction jumped to 4.70. This goal was highly supported by our changes.

15 of 20

Usability vs. Performance Tradeoffs

Figma limitations
No real errors
Figma on desktop ≠ real experience

Might underestimate real-world usability
Good design validate, testing environment limited insight
Suggests potential is strong, further live testing needed

Trade Offs

Impact on Product Market Potential

Since we built our prototype in Figma, there were a few key limitations. First, not everything was clickable. Users were forced to follow a specific path, which meant we couldn’t simulate open-ended exploration—or real-world confusion. Second, we couldn’t simulate or allow actual errors to happen. That means our testing didn’t capture how users might make mistakes and whether the system helps them recover. So, the error prevention results weren’t a perfect one-to-one comparison. Finally, using Figma in a desktop browser felt less intuitive for some users compared to a real interactive website or a mobile app. While our data suggests the redesign is much more usable, we recognize that a real implementation might surface new challenges.�That said, we believe the potential is strong, and that further testing in a live environment will only help fine-tune the user experience.

1 of 20

2 of 20

3 of 20

4 of 20

5 of 20

6 of 20

7 of 20

8 of 20

9 of 20

10 of 20

11 of 20

12 of 20

13 of 20

14 of 20

15 of 20

16 of 20

17 of 20

18 of 20

19 of 20

20 of 20