Issue Severity Subgroup

​

TPAC presentation, 13 September 2022

1

Subgroup members

• Alastair Campbell
• Francis Storr
• Sarah Horton
• Shawn Thompson
• Todd Libby

​

Sub-group wiki page

2

Presentation objectives

• Share prototype
• Decide on whether to continue to explore approach
• If yes, decide on next steps
• Understand dependencies

3

Outline

• Question we’re trying to answer (core questions)
• Prototype: Severity Assessment
• Approach: Post-testing evaluation
• Next steps
• Discuss dependencies
• Appendix: Barrier walkthrough method

4

Our core questions

• How can we ensure WCAG 3 better reflects the "lived experience" of people with disabilities on the web?
• How can we make conformance apply to larger / fast changing web sites and applications?
• How can we ensure WCAG 3 conformance is achievable for smaller organisations and individuals, without setting the bar too low for larger organisations?
• How do we avoid requiring pages pass 100% of the tests but still ensure accessibility?
• How do we improve the equity of the guidance between different disability types, now and in the future, while preserving a reasonable effort bar for conformance?
• How do we ensure that with issue tolerance, we do not allow conformance with critical / blocking errors?"

5

Approaches discussed initially

• The scoring & critical issues from the WCAG 3 FPWD�
• Issue severity matrix�
• Barrier score�
• Barrier walkthrough method�
• Usability testing

6

Severity Assessment Prototype

• Break down several guidelines to establish severity at the test level.�

​

7

Severity assessment mapping

8

Severity assessment notes

• User categories are a less-detailed equivalent to WCAG’s Functional Needs.
• We would use WCAG’s Functional Needs.

9

Severity assessment usage

Positives:

• It appears to be possible to rate and map each test.
• Could show who is impacted and how, per guideline/method.
• With more work on levels, provides a more aligned basis for scoring.

Considerations:

• Requires all tests to be evaluated for impact across functional needs.
• Requires you to review all instances of failures.
• Some form of scoring is needed for it to be used in conformance.
• How do you differentiate critical from non-critical failures?
• No assessment of impact on task

10

Counting and assessing content (from 2020)

• Jeanne and Francis worked on this last year and Jeanne recently presented on it.
• The feedback wasn’t overly positive, especially around the need to count things (for example: all the images in a user journey).
• If you want to read up on this approach, read the Evaluating the WCAG3 Proposal deck

11

Approach: Post-testing severity evaluation

• Do all the testing first, then assess how big a barrier each instance is, in context.
• Per “scope” of the conformance statement, need to identify all known issues;
• Walk through an assessment of severity based on:
• Impact derived from guideline/method/test (e.g. A vs AAA, or severity assessment)
• Impact on task (need to identify tasks)
• Functional needs impacted;
• (Can include organisation oriented things like which component is impacted, how big a task it is to fix etc.)
• Needs to be a process that happens once issues are identified;
• Could contribute to conformance, but not as part of the guideline/method layer.

12

Post-testing severity evaluation

Positives:

• Can account for task impact, a key factor from usability testing.
• Could be an optional extra to improve your score.
• Very useful for prioritisation.

Considerations:

• Requires you to review many instances of failures.
• Some form of scoring would be needed to use it in conformance.
• Requires knowledge of typical usage (e.g. of AT), unless a lot of info is provided to assist the evaluation.

If this seems like a good choice, we could draw from the barrier walkthrough method.

13

Issue Severity opportunities pull request

Content added to the Opportunities section (rawgit preview)

​

14

Issue Severity Functional Images method pull request

15

Content added to the Functional Images method (rawgit preview)

Issue Severity Translates Speech method pull request

Content added to the Translates Speech method (rawgit preview)

16

Next steps

• Document any feedback from the group
• Talk about next steps

​

17

(Appendix) Barrier walkthrough method

18

Overview: Barrier Walkthrough Method

“The basic underlying idea [of the Barrier Walkthrough Method] is that, for testing and assessment purposes, it is better to start from known types of problems rather than using general design guidelines. (This is the same approach you would follow when assessing security of a web site: you'll start from known vulnerabilities.)”

“A barrier is any condition that hinders the user's progress towards achievement of a goal, when the user is a disabled person. A barrier is described in terms of:

• the category of user and the type of disability
• the type of assistive technology being used
• the failure mode, that is the activity/task that is hindered and how it is hindered, and
• which features in the page raise the barrier.”

19

Barrier Walkthrough Method: user goals

• User goals are use cases for people using the web site.
• For example: “create a new account”, “buy a set of footwear for hiking in Scotland in December”, “create a new issue to log a bug you’ve discovered”.
• This is similar to WCAG-EM’s Identify Essential Functionality Of The Website step.
• Once use cases have been identified, possible user journeys are identified. For example:
• The shortest possible successful journey
• A journey where the user has forgotten their password

20

Barrier Walkthrough Method: possible barriers

Possible barriers include accessibility issues such as:

• Functional images lacking equivalent text
• Video with no captions
• Color is necessary
• Functional images as background images
• Generic links
• Ambiguous links
• Mouse-only events
• Keyboard traps
• Form elements with no accessible name

21

Barrier Walkthrough Method: performance variables

Consequences of barriers should be described using performance variables:

• effectiveness: the ability to achieve a given goal accurately (execution errors, mistakes or slips), and completely (quality of the solution);
• productivity: the resources (time, effort, cognitive load) that are needed in order to reach a certain level of effectiveness;
• satisfaction: acceptability and pleasure of use (frustration, perceived ease of use, perceived productivity, perceived security);
• security: personal safety and financial security.

​

22

Barrier Walkthrough Method: minor problem

The barrier is detected by the user, but there are simple ways to overcome it or to avoid it; it is easy to remember it, to learn how to avoid or get around it. This barrier affects marginally productivity or satisfaction, but not effectiveness nor safety.

23

Barrier Walkthrough Method: significant problem

The barrier is detected and it heavily affects the task execution. To overcome the barrier the user has to back-up, follow a trial-and-error strategy, guess the proper action, repeat an action several times; the user may incur in errors. In many cases it is not possible to avoid the barrier, which reduces effectiveness and/or security; even if it can be avoided, this requires a substantial knowledge and/or memory (to recall that there is the barrier and on how to avoid it). The barrier affects effectiveness, productivity, satisfaction and also safety.

24

Barrier Walkthrough Method: critical problem

The barrier is so big that very often users give up, and they do not reach their goals. This can happen after users have spent considerable time and effort to try to overcome the barrier, perhaps with many errors. There are no alternative ways (known to the users) that can be followed to achieve the goals. The barrier has a strong negative impact on effectiveness, and consequently also on productivity, satisfaction and safety.

25

Barrier Walkthrough Method: procedure

1. Select the user categories / functional needs
2. Select a user goal
3. For each page in the goal, assess whether a barrier is present
4. If a barrier is present, determine the:
1. severity of the barrier in relation to the user being able to successfully complete their goal.
2. user performance factor that is mostly affected by the barrier.

A single defect in the page may raise more than one barrier.

26

Barrier Walkthrough Method: assigning a score

• Use a scale of 1 to 3 (3 is the worst case) for each of the two parameters.
• Use the following table to assign the severity score, and classify the problems as critical (score =3), significant (score = 2) or minor (score = 1).

27