1 of 39

Beta Test 4 Insights

VAC Find Benefits & Services

March 2019

2 of 39

Test

Snapshot

3 of 39

3

2

3.5

Facilitators

Notetakers

Practice tests

Days of testing

3

4 of 39

Test focus

Do users find the tool to be a useful introduction and entrypoint to VAC benefits and services?

Are users able to find relevant benefits to meet their needs?

Do users know and plan what their next steps are upon leaving the tool?

Does the tool meet the expectations of mobile users?

Methodology

Task completion

Qualitative feedback

Confidence level

Comprehension

The test focus was very much the same as the previous tests, but we did conduct half of our tests with an iPad, and we tested the VAC homepage design to see where people would go to look for information, and if the FB&S CTA was effective.

Our goal is to:

- see if users understand what they need to do and can complete all relevant tasks

- identify specific usability issues - for example, problems with the language or layout.

- generate ideas for how to improve our service.

Concurrent Think-Aloud

Users articulate their thoughts and reactions as they progress through the website, completing the task specified.

Rationale

Encourage participants to keep a running stream of conscious thoughts as they work.
Understand participants’ experience of the interface as they interact with the website
Elicit real-time feedback and emotional responses
Does not require recording of participant on-screen actions or voice

https://www.gov.uk/service-manual/user-research/using-moderated-usability-testing

5 of 39

31

We conducted tests with 31 different users

6 of 39

Petawawa

10

4

13

3

Transition Centre
Gym
MFRC
Employment Services

Veterans

Releasing members

Still-serving members

Family members

7 of 39

Usability

Insights

8 of 39

Usability issues

Expecting multiple selections in guided experience
Releasing members identifying as still-serving
Flow is not intuitive for family members

9 of 39

1

Expecting multiple selections in guided experience

10 of 39

Users asked if they could select multiple options, or expressed that they would want to.
The consensus among users is that they would want to see results for both them and their family.

11 of 39

2

Releasing members identifying as still-serving

12 of 39

“As of now, or when I’m retired?”
“I know I would have access to more benefits once I release.”
“Looking to get medically released.”

This was a big one. Those users who identified on our screener that they were releasing, along with those who told us they were looking to be released selected still-serving on the first GE question. This is an issue because they aren’t seeing results that will be relevant to them upon actually releasing.

People seemed to identify more as still-serving than as a Veteran as they were not yet released. Through talking to people at the Base during our trip, it was interesting to hear first-hand that many people have trouble accepting that they are no longer serving (one staff member told us that his VAC papers have been sitting on his desk for a year, and even though he preaches to others to apply, he said that doing so himself would be accepting that his career in the military is actually done).

Now that the still-serving option is removed, it will be interesting to see with additional usability testing of releasing/transitioning CAF members if this issue still exists, or if new issues related to it are identified.

13 of 39

3

Flow is not intuitive for family members

14 of 39

Family member (spouse) takes care of things, including benefits for serving/Veteran spouse.
Service-related GE question didn’t make sense to one user.
Some wording on benefit cards was unclear as to how it relates to family members (ex: Vocational Assistance).

15 of 39

Additional usability issues

Expecting a connection to MVA
Unsure where to find “Do you qualify” information
Confusion around categories and benefits
Confusion around April 1, 2006 release date
Looking for specific words that were not found

16 of 39

1

Expecting a connection to MVA

17 of 39

2

Unsure where to find Do you qualify information

18 of 39

3

Confusion around categories and benefits

Next was the confusion around categories and benefits and how the two relate. Some users thought that the categories were the benefits (particularly Emergency Funds and Education and Training, as they are so close to the associated benefit names). Some users thought it was odd that benefits would show up under more than one category. And some users were unsure how the benefits related to the categories (one specific instance of this was a user looking at Rehab on the results page, seeing the Education and Training category tag, and looking for information related to education and training in the card content).

The updated logic pathways may clear some of this up, and some if it may be related to the amount of knowledge that some users already have about VAC services, but it might be worth taking a more in-depth look at some of the category names.

19 of 39

4

Confusion around April 1, 2006 release date

20 of 39

5

Looking for specific words that were not found

21 of 39

Additional

Insights

22 of 39

Observations

Most users are not scrolling far enough to see the Find benefits and services CTA
Many users would go straight to MVA
Most users navigate the tool using the back button

The first two observations here are related to the entry to the tool from the VAC homepage.

The majority of users, when presented with the VAC homepage, indicated they would click through the categories on that page to find what they were looking for. Most of them did not scroll far enough down the page to see the Find benefits and services CTA. As discussed at our sprint review on Friday, this is not really a usability issue, and speaks well to how the VAC homepage has been redesigned. However, it is still worth noting, and we should continue to monitor this homepage activity alongside the Communications team via analytics.�
A number of users also indicated that they would go directly to My VAC Account and log in. These users typically had some familiarity with VAC already, but this was interesting to note as we at VAC work towards all releasing members creating an account as part of their release/transition process.�
As far as navigating around the tool itself, and specifically back through the guided experience questions, most users instinctively used the back button to do so (even when not asked to change a response to a GE question). While this was by far the most used method, a number of users also used the Home breadcrumb. We’ll need to be sure that the breadcrumbs are clear and functional, as some people used the Home breadcrumb on the program page, expecting to get back to the tool, but ended up on the VAC homepage.

23 of 39

Next steps

Would talk to VAC
Would research further on program pages
Would talk to friends/loved ones

These are the top 3 next steps that people indicated throughout testing

The number one next step was that users would talk to VAC. Some identified that they would call, some would visit an area office, but the common thread was that they would take what they learned through using the tool and researching information, and speak about it further with someone from VAC.�
There was a tie for number 2… Some users indicated that they would continue to research more, identifying the program pages within the website as the focus.�
And just as many users indicated that they would talk to friends or family. This included family members sharing information with their spouse, and sharing information with friends in the military, who may or may not have knowledge about VAC services.

The next common answers were to log into MVA, talk to an organization other than VAC (specifically, MFRC and RCL were mentioned), and only 2 people said they would apply at this stage. This was interesting to me because so many people in the last round of testing said that they wanted to apply. However, the majority of testers this time around had very little knowledge of VAC and its services, so the need for additional research and guidance made sense.

24 of 39

Family Members

Family members are often looking for information about benefits for their spouse.

25 of 39

Suggestions

26 of 39

Releasing should be an option in the guided experience.

27 of 39

Replace Death & Bereavement With

Loss of life

28 of 39

Service-related health issue should have PTSD, anxiety, and addiction as examples.

29 of 39

Days are calculated differently for reservists than for regular force – this should be noted somewhere.

30 of 39

Quotes

31 of 39

“

I wouldn’t sit at a computer (myself); I’d have someone go through it with me.

32 of 39

“

I would like to know more, but I don’t even know how to start, what to ask, what to do.

33 of 39

KPIs

34 of 39

Key performance indicators

Increased findability
Increase in perceived confidence
Decrease in unclear words

35 of 39

Let’s take a look at the spreadsheet!

36 of 39

Results

Did you find what you were looking for?

Yes - 23 (77%)
No - 7 (23%)

How confident are you from 1-10 that you found information relevant to you?

Average of 7.4

Are there any words that are unclear?

7 words identified by 5 users

Previous results

Ottawa + Gatineau (17 users)

Yes - 9 (53%)
No - 8 (47%)

Average of 5.7

8 words identified by 5 users

37 of 39

Want all the nitty gritty details?

Find our detailed rainbow analysis and additional notes and observations on Google Drive.

38 of 39

Next steps...

Decision dashboard
Ideation and prioritization
Plan for additional usability testing of new design

39 of 39

Thank you!

Questions?