1 of 23

Member Portal Baseline Test

Research Findings Presentation

March 2022

TM

2 of 23

Background & Objectives

Proprietary & Confidential

2

TM

3 of 23

Background

We want to baseline and validate the key user journeys in the member portal. In the future as we make updates and enhancements, we can measure the effectiveness of changes by benchmarking against these findings.

​

Problem Statement

What is this research about?

Proprietary & Confidential

3

  • UX Designers: Grace Deck, David Mclean
  • Product Owners: Kristina Krautbauer, Chris Lamb

​

Stakeholders

Key Dates

03/09/2022

Kick-off

03/23/2022

Pilot

03/23/2022

Conduct study

03/24 – 03/30

Analysis

03/31/2022

Read out

TM

4 of 23

Objectives

What problems do we hope to solve?

Proprietary & Confidential

4

KPI

Metric

Task performance

  • Task time
  • Task completion rate
  • Error/Failure rate

Perception of ease, task level satisfaction

Single Ease Question (SEQ)

Perception of usability, test level satisfaction

System Usability Scale (SUS)

TM

5 of 23

Objectives

  • Understanding the usability of the core journeys in the member/patient portal
  • Perception of ease, usefulness, satisfaction
  • Attitudes towards visual appearance

​

Success Criteria

Why are we doing this research?

Proprietary & Confidential

5

TM

6 of 23

Research Scope & Methods

Proprietary & Confidential

6

TM

7 of 23

Research Methods

Unmoderated quantitative usability test utilizing user zoom go platform, to produce a summative evaluation to assess the overall performance of the member portal. While also capturing qualitative insights from recorded sessions to help produce a formative evaluation based on key insights.

Test will use a clickable prototype with screens being as close to the actual experience as possible, but with limited pathways resulting in dead ends.

Link to prototypes

​

Primary Method

How did we execute this research?

Proprietary & Confidential

7

Secondary Method

15

90% confidence with ± 24% margin of error

Participants

TM

8 of 23

Key Findings

Proprietary & Confidential

8

TM

9 of 23

Perception of Usability

Test level satisfaction

Proprietary & Confidential

9

  • Simple ten-item attitude Likert scale giving a global view of subjective assessments of usability
  • Provides high level subjective view of usability and often used in carrying out comparisons of usability between systems (this sets us up for if we ever want to benchmark against competitor sites in usability tests)
  • https://en.wikipedia.org/wiki/System_usability_scale
  • https://stuart-cunningham.github.io/sus/

​

More about SUS

95

�System Usability Scale (SUS) Score

​

By conventional standards, a score of 95 is considered to be above average.

�This score is considered a Grade A.

�This score may be described as best imaginable.

TM

10 of 23

Task 1: Create an account

Task level satisfaction

Proprietary & Confidential

10

  • “The process of signing up and creating an account was intuitive and simple. These things made the task very easy to complete.” – Participant 5133863
  • Several participants missed the “Sign up now” link to create a new account and stopped 1 participant from completing the task
  • 1 participant was concerned about security when no password requirements were shown

​

Notables:

Metric

Result

Task time

1:56

Task completion rate

88%

Task error/failure rate

12%

Single Ease Question (SEQ)

6.50 out of 7

TM

11 of 23

Task 2: Member story

Task level satisfaction

Proprietary & Confidential

11

  • “The questions were straightforward and easy to understand. The radio button functionality was intuitive.” – Participant 5133869
  • “Lot of questions but all were important” – Participant 5133928
  • 1 participant was frustrated by the link title “Tell us your experience” and would’ve rather it said something like “Get started”
  • 1 participant found referring to the survey as “Member story” in the success message odd

​

Notables:

Metric

Result

Task time

1:51

Task completion rate

100%

Task error/failure rate

0%

Single Ease Question (SEQ)

6.75 out of 7

TM

12 of 23

Task 3: Upload a bill

Task level satisfaction

Proprietary & Confidential

12

Metric

Result

Task time

0:35

Task completion rate

100%

Task error/failure rate

0%

Single Ease Question (SEQ)

6.88 out of 7

  • “Very Easy- I knew where I needed to click to upload the bill and the upload process was very smooth.”– Participant 5133887
  • “It was what I am used to seeing on other sites for uploading documents” – Participant 5133927
  • 1 participant felt like response should be quicker than 2-3 business days

Notables:

TM

13 of 23

Task 4: Sign forms

Task level satisfaction

Proprietary & Confidential

13

Metric

Result

Task time

1:20

Task completion rate

81%

Task error/failure rate

19%

Single Ease Question (SEQ)

6.25 out of 7

  • “If Adobe Sign is easy to go through all the places to sign, it is pretty easy. Its nice that it seems like you can sign both forms in on link” – Participant 5134167
  • “The email pointed out the specific forms that I would need to sign and linked me over to the form signature area.” – Participant 5133887
  • Due to this being a clickable prototype, I think some folks got frustrated with not having all the Adobe Sign functionality they are familiar with.�

Notables:

TM

14 of 23

Task 5: Account Settings

Task level satisfaction

Proprietary & Confidential

14

Metric

Result

Task time

0:30

Task completion rate

100%

Task error/failure rate

0%

Single Ease Question (SEQ)

6.81 out of 7

  • “The setting/account info at the top was very clear and it was very easy to navigate” – Participant 5134167
  • “Account management seemed the same as every other website ive used” – Participant 5134130
  • “Changing my email was exactly where I expected it to be.” – Participant 5133927��

Notables:

TM

15 of 23

Task 6: Email your advisor

Task level satisfaction

Proprietary & Confidential

15

Metric

Result

Task time

0:27

Task completion rate

94%

Task error/failure rate

6%

Single Ease Question (SEQ)

6.75 out of 7

  • “Multiple pathways to connect and send correspondence to my advisor made this easy. All form based and simple.” – Participant 5133857
  • 1 participant missed the advisor contact link in the case card, “There wasn't a link directly on my case, so I had to navigate to the right to see where to email my advisor.” – Participant 5133927

​

Notables:

TM

16 of 23

Task 7: Negotiation outcomes

Task level satisfaction

Proprietary & Confidential

16

Metric

Result

Task time

1:21

Task completion rate

81%

Task error/failure rate

19%

Single Ease Question (SEQ)

5.75 out of 7

  • At least 9 participants, a little over 50%, said they would like to see their final document on the case card vs. having to navigate to the “Documents” tab in the header
  • “Why wouldn’t you have a link from here showing that as opposited to putting it documents? You can put it there too, but there should be something called, you know, outcomes, that’s wrong. Ok you blew this one” – Participant 5134029

Notables:

TM

17 of 23

How to use these insights going forward

Proprietary & Confidential

17

TM

18 of 23

Scorecard Template

Proprietary & Confidential

18

KPIs

Baseline Results

{Future Test Results}

Analysis

Task time

1:21

​

​

Task completion rate

81%

​

​

Task error/failure rate

19%

​

​

Single Ease Question (SEQ)

5.75 out of 7

​

​

  • Make the test as similar as possible to the baseline test
  • Include SEQ and follow up explanation in after task questions
  • Only measure and benchmark SUS when testing against all journeys included in the baseline test

TM

19 of 23

Appendix

Proprietary & Confidential

19

TM

20 of 23

Screener results

Proprietary & Confidential

20

TM

21 of 23

Screener results

Proprietary & Confidential

21

TM

22 of 23

Screener results

Proprietary & Confidential

22

TM

23 of 23

Screener results

Proprietary & Confidential

23

TM