Task and feedback analysis:

Financial advice and support tasks

November 23, 2020

Digital Transformation Office - Canada.ca

​

​

UNCLASSIFIED / NON CLASSIFIÉ

• DTO was asked to take a closer look at the Financial support group of tasks reported in the COVID top task exit survey
• We analyzed October feedback from the two DTO-managed research tools:
• COVID top task exit survey,
• User feedback tool placed on a small number of pages�(Economic Response Plan, Support for businesses)
• Share some of the concerns and successes expressed through these feedback mechanisms as well as some preliminary recommendations

​

PURPOSE

UNCLASSIFIED / NON CLASSIFIÉ

EVIDENCE-BASED CONTINUOUS IMPROVEMENT

Call centre data

Usability testing

- Pilot COVID top task survey to understand performance

​

- Use insights from user feedback tool to guide improvements needed

Web analytics

PRODUCT DEVELOPMENT

PRODUCT IMPROVEMENT

LAUNCH

OPTIMIZATION

Exit surveys

User feedback

UNCLASSIFIED / NON CLASSIFIÉ

HOW ARE THE COVID TOP TASKS PERFORMING?

Based on October data (See Annex for September data)

*confidence in this success rate may have a large margin of error due to <384 responses

UNCLASSIFIED / NON CLASSIFIÉ

• Covers COVID-specific financial tasks, but has overlap with tasks that exist outside of the COVID context.
• Survey invitations are only placed on COVID-specific pages - not all financial support pages.
• The task success metric requires 384 responses to have a 95% confidence within +/- 5%. For smaller tasks, we use more than a month of data to gain sufficient response numbers to analyze.

​

ABOUT THE COVID TOP TASK SURVEY

UNCLASSIFIED / NON CLASSIFIÉ

HOW WE USE THE COVID TOP TASK SURVEY

• Collect write-in feedback (a feature not yet available in the CX exit survey)
• Help us prioritize our work
• Gain preliminary insights about issues within these tasks
• The COVID top task survey should be used in conjunction with the Principal Publisher CX exit survey and other points of data (call centres, help lines, user feedback) to fully understand pain points

UNCLASSIFIED / NON CLASSIFIÉ

Financial advice and support tasks measured in the COVID-19 exit survey

​

​

• Canada Emergency Response Benefit (CERB) (Program ended and has been removed)
• Recovery benefits (self-employed, caregiving, sickness)
• Employment Insurance (EI)
• Canada Emergency Wage Subsidy (CEWS)��
• Support for businesses
• Support for individuals
• Support for people with disabilities
• Support for seniors

Significant changes to benefit programs for individuals and businesses

• CERB ended and Canadians transitioned to EI or the new CRB
• Eligibility for EI expanded (one-time credit of insured hours, unemployment rate changes)
• CRCB and CRSB started
• CEWS program changed
• CECRA ended and CERS was announced but was not yet opened

For individuals, transitioning from CERB meant people needed to learn new processes to continue getting the support.

​

Likewise for CEWS, changes to calculations meant business people needed to relearn how to calculate their subsidy.

​

We expect October’s success rates to be lower for these tasks due to these changes. Looking at November data will provide more insight and help identify any trends.

​

OCTOBER WAS A MONTH OF CHANGE

UNCLASSIFIED / NON CLASSIFIÉ

Employment Insurance tasks overview

Section summary:

• COVID top task ranking: 4 (No change from September)
• Overall success rate: 66.20% (Down 10% from September)
• Submit EI report: 78%
• Check your eligibility for EI, Apply for EI, Check status of a payment: �< 384 responses
• Unique visits: 2,142,714 visits to the landing page in October
• Devices used: 45% Mobile

​

Top feedback topics:

• Transitioning from CERB to EI (An October-specific issue)
• For people on EI: extensions / what to do if EI ran out / reactivating a claim
• What program to apply for? Eligibility issues
• Account support

What’s working:

Once people understand how to do bi-weekly reporting, success for that sub-task is high (78% in Covid exit survey, 84% in CX exit survey)

​

Actionable recommendations:

1. Make the bi-weekly reporting link to be easier to find (7 comments)
2. Make it easier to get account assistance from the internet reporting page.
3. Gather research from CX exit survey, product owners, call centres, to understand pain points. This task extends beyond the COVID context.

Success rates are trending downwards for this task.�When CERB ended, many Canadians needed to switch to a more complicated EI process

*confidence in this success rate may have a large margin of error due to <384 responses

UNCLASSIFIED / NON CLASSIFIÉ

CEWS task overview

Section summary:

• Section url: CEWS
• COVID top task ranking: #5 in Financial support tasks
• Success rate: 67.7%* since July (338 responses)
• Unique visits: 427,990 Visits to the landing page in October
• Devices used: 78% Desktop

​

Top feedback topics from user feedback:

• Eligibility
• Calculations (calculator, spreadsheet)
• Applying
• Page issues

What’s working:

CRA added the feedback tool on all CEWS pages to monitor feedback on September 28..

• As at November 11:
• 16,617 Responses Received
• 59% found what they were looking for
• 1,494 Comments Received
• 1,162 English / 332 French Comments �

Actionable recommendations:

The most relevant comments related to the “Calculations (calculator, spreadsheet)” theme, were about “revenue drop”. The UX team should consider testing tasks related to this calculation.

​

Looking at comments from the page feedback tool will be key to understanding user frustrations

*confidence in these success rates may have a large margin of error due to <384 responses

UNCLASSIFIED / NON CLASSIFIÉ

Recovery benefits task overview

​

Page summary:

• Includes: CRB, CRSB, CRCB
• COVID top task ranking: #2 in Financial support (new task in October)
• Success rate: 59.67%* (181 responses - no previous data)

​

Top feedback topics:

• Many questions looking for how and where to apply during gap between announcement and launch of application process - this may have brought the success rate down
• Technical issues when the program launched may have also contributed to a lower success measure

What’s working:

• Page feedback widget placed on all the CRA pages for recovery benefits to gain specific insights
• CRA performs regular usability testing prior to launch and program changes

​

​

Actionable recommendations:

Avoid delays between announcements and program availability (CERB ended September 27, two weeks before CRB applications opened)

​

*confidence in these success rates may have a large margin of error due to <384 responses

UNCLASSIFIED / NON CLASSIFIÉ

Support for individuals task overview

​

Page summary:

• Page url: No specific URL
• COVID top task ranking: #4 in Financial support tasks
• Success rate: 55.4% since July (428 responses)

​

Top feedback topics:

• People were waiting for the new Recovery Benefits to open and figure out if they were eligible
• People were unsure what program(s) they may be eligible for
• Reporting technical issues encountered when applying for benefits
• Some groups of people may be falling through the cracks �(single parents, low income earners, part time employees)

What’s working:

1. Access to assistance by phone

​

Actionable recommendations:

• Announce new supports only once they are available for application or once the eligibility criteria, support amounts and application details are confirmed and an application launch date is known and communicated to the public;
• Encourage use of benefits finder from key pages

​

​

Not enough data to present monthly trends

​

UNCLASSIFIED / NON CLASSIFIÉ

Support for people with disabilities task overview

​

Page summary:

• Page url: No specific URL
• COVID top task ranking: #6 in Financial support (up from September)
• Success rate: 48.8%* since July (160 responses)

​

​

Top feedback topics:

• After the disabilities benefit was announced, many were looking for updates on when the payment would be sent
• Like other groups, people weren’t sure what programs they were eligible for
• Some were looking for more funding for this group of people

What’s working:

• Additional disability financial aid

​

​

Actionable recommendations:

1. Make it easier to identify the payment reason in direct deposit statements. People were unsure why they had money deposited.
2. Provide clear dates for when the disability support payment would be deposited
3. Gather research from CX exit survey, product owners, call centres, to understand pain points. This task extends beyond the COVID context.

​

​

​

*confidence in these success rates may have a large margin of error due to <384 responses

UNCLASSIFIED / NON CLASSIFIÉ

Support for seniors task overview

​

Page summary:

• Page url: No specific URL
• COVID top task ranking: #8 in Financial support (no change from September)
• Success rate: 56.2%* since July (105 responses)

​

​

Top feedback topics:

• Seniors generally feel overlooked when it comes to financial support
• Looking for guidance on what to apply for
• Looking for specific support for low income / disabilities

What’s working:

• Additional financial aid for seniors
• ​

​

Actionable recommendations:

1. Make it easier to identify the payment reason in direct deposit statements. People were unsure why they had money deposited.
2. Gather research from CX exit survey, product owners, call centres, to understand pain points. This task extends beyond the COVID context.

​

​

​

*confidence in these success rates may have a large margin of error due to <384 responses

UNCLASSIFIED / NON CLASSIFIÉ

Support for businesses task overview

​

Page summary:

• Page url: Covers multiple sections
• COVID top task ranking: #7 within Financial support (no change from September)
• Success rate: 59.5%* since July (121 responses)

​

Top feedback topics:

• Spikes in comments after announcements ( Black entrepreneurship program or new rental subsidy program)
• Difficult to determine what programs to apply for
• Growing need for more information on the new rental subsidy program
• Most feedback now pertains to accessing financial assistance rather than guidance on reopening

What’s working:

• Program changes to CEBA (bank accounts, extension)
• Programs that are grants work better than loans for small business

​

​

Actionable recommendations:

1. Avoid delays between announcements and program availability
2. Include dollar amounts in program descriptions - feedback often refers to loans by the amount rather than name - “$20,000 loan”

​

*confidence in these success rates may have a large margin of error due to <384 responses

Not enough data to present specific trends

​

UNCLASSIFIED / NON CLASSIFIÉ

Announce new supports only once they are available for application OR once enough eligibility criteria and launch date can be communicated to the public

• CERB ended September 27, but Recovery benefits opened 1-2 weeks later �(CRSB/CRCB opened on October 5, CRB applications opened October 12)
• CERS announced October 9, applications opened November 23
• Black entrepreneurs funding announced September 9, application/process not yet open�

Avoid using news releases as web content

• Announcements should point to web content that is written in plain language with program details.

Common issues to avoid

UNCLASSIFIED / NON CLASSIFIÉ

Addressing low-response tasks�General tasks like Support for individuals, businesses, seniors, people with disabilities have low monthly responses. It may take 4-6 months to gain the minimum number of responses to analyze.

​

Combine low-response tasks into a single task�We think it is better to revise these general tasks into a single task: “Find financial support that I can apply for”.

These audiences share an overlapping goal of finding financial support.

Pros:

• A bigger sample size of responses per month
• Users can still add feedback that will help us identify needs by audience

​

Changes to the COVID exit survey for December

UNCLASSIFIED / NON CLASSIFIÉ

COVID top task exit survey resources

​

These resources are posted on the GC Wiki under “User research”

• COVID top task survey monthly analysis
• COVID top task survey written feedback (updated every 2 weeks)

​

CX exit survey (managed by the Principal Publisher)

​

Data from the CX exit survey is available through your Adobe Analytics account.

UNCLASSIFIED / NON CLASSIFIÉ