1 of 19

Homepage Redesign

& JOA Exploration

Usability Testing Plan

2 of 19

We plan to conduct 2 separate tests in one session.

Homepage Redesign

Interview oriented usability test

OBJECTIVE: Determine what final changes are needed to improve the redesign before its development.

JOA Exploration

Comparative usability test

OBJECTIVE: Determine which elements of two JOA designs (current and exploratory) work best and should be implemented.

​

Then proceed to

1

2

3 of 19

Why combine two separate tests?

  1. To save time and energy
  2. Both tests have similar target audiences
  3. To recruit participants once

4 of 19

Homepage Redesign

Usability Testing Plan

Part 1

Approximately 25 minutes

5 of 19

Purpose and Scope

Background

The homepage redesign is part of USAJOBS’ Design System 3.0 upgrade work, making any content and design improvements of the website along with the upgrade. As USAJOBS’ website design has not changed in the last 6-7 years, the USAJOBS team reviewed the user’s pain points when using the website and looked at google analytics data to identify outdated content and areas to improve.

​

After a series of design iterations reviewed by the PO, BA’s, and Content, the following 7 areas of the homepage were improved in the redesign:

  1. Search bar and banner area
  2. Featured branded search page
  3. Module tabs
  4. Hiring path links
  5. Events
  6. Help Center links - Federal hiring myth
  7. Footer

The objective of this test is to evaluate the homepage redesign and determine any other final changes needed to improve it before its development by answering the following questions:

  1. What are the target user’s initial impressions of the redesigned homepage?
  2. Does the redesigned homepage explain what USAJOBS is and what its services are? 
  3. Do visitors understand how to use the page’s widgets to get started with their job search, create a profile, find events related to their job sector, etc.?
  4. Is the new design’s mobile experience user friendly? 

JOA Exploration

6 of 19

Audience

As USAJOBS targets any visitor looking for a federal job, the study will be conducted with a minimum of 18 participants from the following audience groups who represent a spectrum of familiarity with the website. Public participants will be limited to a total of 9 for PRA concerns, and the same number or more federal employees will be recruited in order to have a better comparison of federal to non-federal employee responses. Participants will be recruited through the existing list of federal employees interested in participating in research projects and an open call posted on LinkedIn.

​

    • At least 9 Federal employees
    • 9 general public people interested in Federal jobs but have no familiarity with USAJOBS

Outcomes

The study will provide us with: 

  • Behavioral performance data.
  • Actionable insights on how to optimize the user experience for visitors.
  • Insights on how to make the Events section more noticeable, if it currently is not, without breaking the information architecture that drives the single most important engagement for the target: finding jobs.

Similar Jobs

Methodology

The remote moderated usability study over Zoom meeting, screen sharing, and recording will collect qualitative and quantitative data using observational and informative test methods to answer the following research questions: 

    • Is the visual hierarchy of the page structured in a way that makes sense? (What part of the homepage are users most drawn to initially, find out of place, or find aesthetically pleasing?) 
    • Can visitors easily digest information provided on the homepage?
    • What pieces of information are visitors expecting to find on the homepage?
    • How likely are the users to read the Events section?
    • How is their experience on mobile?

7 of 19

Schedule

We will conduct this remote user test within two months.

Team

  • Yuki Shimano

​

Possible members: 

  • (notetakers)
  • (Recruiters)
  • (Test scheduler)

Contingencies

Timeline and outcomes are contingent on: 

  • Team availability to review materials, set up a test system, lead study recruiting and assist with conducting the study. 
  • Availability for walk-through and reviews. 
  • Recruitment of usability test participants. 
  • Approval from product owner (John).

Similar Jobs

Jan 7

Usability Test Begins

Recruiting Begins

Synthesize Findings

Dec 21

Jan 27

8 of 19

Public Participant Screener Questions:

Potential test participants from the general public will be asked the following screener questions in order to determine their eligibility for the usability test.

​

  1. Which of the following best describes your employment status?
    • Full-time
    • Part-time
    • Seeking opportunities
    • currently Retired (disqualified)
    • Prefer not to say (disqualified)
  2. Have you ever applied for a federal job?
    • Yes (disqualified)
    • No
  3. Have you looked for opportunities to work for the federal government in the past year?
    • Yes
    • No (disqualified)
    • Considering

Similar Jobs

9 of 19

Pre-Session Questions

Participants will be asked the following questions before starting the usability test tasks to understand more about their experience with job searching. �

  1. Tell me a little about your current role.
  2. Are you familiar with the job searching on USAJOBS? If yes, tell me about your experience? How often do you search on USAJOBS?
  3. Have you looked for a federal job in the last year?

Similar Jobs

10 of 19

Scenario and Tasks

Tasks that review whether people respond well to the look/feel of the new page and go to the right place quickly were created. In order to test the mobile experience of the homepage, half of the participants will complete the following tasks on a mobile prototype, while the other half do the same on a desktop prototype.

​

Similar Jobs

Design shown to participant

​

Tasks/ Post-task questions

Observations / metrics to track

Task 1: Scroll about the page. What are your overall impressions?

  • What do you think about the colors of the page?
  • What do you think about the graphics?
  • Is there anything that stands out at you?
  • What do you think this website is for?
  • Thoughts on look and feel of page
  • Adjectives to describe the page
  • Purpose of website in the user’s words

​

Task 2: Where would you go to find open jobs?

  • Besides that method(s) are there other ways to find jobs from this page?
  • Tell me what you think this (Career spotlight/Hiring Paths) section is about?
  • From a scale of 1-5, 5 being very easy, how would you rate the easiness of finding job resources?
  • Time it takes to find where to search for jobs
  • Successful recognition of methods other than search tool (career spotlight section, hiring paths, etc.)

​

Task 3: Find a career fair or networking opportunity that you would be interested in attending.

  • How likely are you to view events when using this page? 
  • When would you look for events?
  • From a scale of 1-5, 5 being very easy, how would you rate the easiness of finding events?
  • Time it takes to find events
  • Situations people would be interested in learning about events and click them

​

Task 4: How would you find the types of jobs that are in high demand of employees?

  • From a scale of 1-5, 5 being very easy, how would you rate the easiness of finding urgent hiring careers?
  • Time it takes to find urgent hiring needs
  • Successful use of accordions

11 of 19

Post-Session Questions

We will ask the following post-test questions to gauge user satisfaction and understanding:

​

  1. Was there anything on the page that you found confusing?
  2. What did you like most about this page?
  3. Did you understand what each section was about?
  4. Was there anything on the page you were expecting to see but didn't?
  5. Did looking at this page improve your understanding of what this website does? If not, what would help you to better understand the purpose of this website?
  6. Is there anything else that you would like to share or discuss?

Applicant Experience

12 of 19

JOA Exploration

Comparative Usability Test Plan

Part 2

Approximately 25 minutes

13 of 19

Purpose and Scope

Background

The JOA exploration started in mid-September and was iterated at biweekly meetings with Felipe Flores, Victoria Clark, and Yuki Shimano. The exploration aimed to visualize the JOA page in a 3-column layout (rather than the current 2-column layout), which is hypothesized to provide more options to organize the content and make it easier for visitors to find information when browsing on a desktop device.

During the iterations of the 3 column layout, the following additional functions that are thought to improve the user experience were added to the design:

  • A sticky and shrinking banner with the job title and apply button
  • A sticky scroll up button at the bottom of the page
  • Putting sections of the content into accordions to collapse information of secondary importance.

​

In order to determine if the “explorative JOA page” (that has content organized in the 3-column layout and new functions derived from the exploration) will improve the user experience , it is necessary to observe how job seekers navigate the page and find information.

The objective is to determine if the new layout and functions of the JOA ideated from the design exploration should be incorporated in the future JOA improvement feature by answering the following questions:

  1. What are the target user’s initial impressions of the 3-column layout JOA page?
  2. How do users find and digest information on the JOA when content is organized into 3 column?
  3. When and how would users use the potential new functions of the sticky banner, scroll up button, and accordions?

​

JOA Exploration

14 of 19

Audience

As USAJOBS targets any visitor looking for a federal job, the study will be conducted with a minimum of 18 participants from the following audience groups who represent a spectrum of familiarity with the website. Public participants will be limited to a total of 9 for PRA concerns, and the same number or more federal employees will be recruited in order to have a better comparison of federal to non-federal employee responses. Participants will be recruited through the existing list of federal employees interested in participating in research projects and an open call posted on LinkedIn.

​

    • At least 9 Federal employees
    • 9 general public people interested in Federal jobs but have no familiarity with USAJOBS

​

Outcomes

The study will provide us with: 

  • Insights on whether the 3-column layout should be used for the JOA or not.
  • Behavioral performance data.
  • Actionable insights on how to optimize the user experience for job seekers.

Similar Jobs

Methodology

The remote study, which will continue after the Homepage Redesign Test with the same participant who shares their screen on Zoom, will identify navigation patterns, potential problems, and design opportunities of the explorative JOA design.

15 of 19

Schedule

Please see schedule provided in Part 1: Homepage Redesign Usability Test Plan.

Contingencies

Timeline and outcomes are contingent on: 

  • Team availability to review materials, set up a test system, lead study recruiting and assist with conducting the study. 
  • Availability for walk-through and reviews. 
  • Recruitment of usability test participants. 
  • Approval from product owner (John).

Similar Jobs

Team

  • Yuki Shimano

​

Possible members:

  • Victoria Clark
  • Susan Fisher
  • Tiffany Rost
  • Jodie Young
  • John Still

16 of 19

Participant Screener Questions & Pre-Session Questions

​

Screener and pre-session questions were already asked in Part 1, so they will not be asked in Part 2.

​

Similar Jobs

17 of 19

Scenario and Tasks

Five tasks that are oriented around the new JOA page 3-column layout and functions aimed to benefit the users were created. Participants will be shown two different designs of the JOA page, and given the following scenario and tasks to complete on their computer or laptop device. The order of which design is presented first will vary randomly by participant.

​

Scenario: Imagine that you searched for a Visual Information Management Specialist job and opened the following job announcement page from the search results.

Similar Jobs

Design shown to participant

Tasks/ Post-task questions

Observations / metrics to track

Task 1: What are your overall impressions of this page?

Task 2: Find out if this job is a full time job or part time job.

Task 3: Find the name of the agency this job belongs to.

Task 4: Find out what documents are required for this job.

Task 5: Find where the anti-discriminatory policies that the application process follows are.

Task 6: Where would you click to apply to this job?

  • On a scale of 1-5 (5 quick), how would you rate how quickly you can find the apply button?

​

  • On a scale of 1-5 (5 easy), how would you rate the easiness to find information on the page?
  • On a scale of 1-5 (5 useful), how would you rate the usefulness of the layout of this page?
  • (Task 1) Thoughts on look, feel and layout of the page
  • (Tasks 2-6) Task completion time

Task 1: What are your overall impressions of this page?

Task 2: Find out if this job is a full time job or part time job.

Task 3: Find the name of the agency this job belongs to.

Task 4: Find out what documents are required for this job.

Task 5: Find where the anti-discriminatory policies that the application process follows are.

Task 6: Where would you click to apply to this job?

  • On a scale of 1-5 (5 quick), how would you rate how quickly you can find the apply button?

​

  • On a scale of 1-5 (5 easy), how would you rate the easiness to find information on the page?
  • On a scale of 1-5 (5 useful), how would you rate the usefulness of the layout of this page?
  • (Task 1) Thoughts on look, feel and layout of the page
  • (Tasks 2-6) Task completion time

18 of 19

Design shown to participant

​

Tasks/ Post-task questions

Observations / metrics to track

III. Version Comparison

As a review, here are the 2 versions of the Job Announcement page you viewed. 

​

Which version do you like best? Explain why.

​

  • The layout/design that is less confusing for users

​

Similar Jobs

19 of 19

Post-Session Questions

We will ask the following post-test questions to gauge user satisfaction and understanding:

​

  1. Is there anything you felt was missing from your experience finding information on the Job Announcement page?
  2. Was there anything in the pages that was particularly confusing? 
  3. Was there anything you liked about the pages?
  4. Are there any other comments, feedback, concerns that you’d like to share?

Applicant Experience