Conformance Proposal

3 Types of Testing�Draft��28 July 2020�Rachael Bradley Montgomery�

1

Overview

• Goals
• Distill Silver conversations to date into a concrete example
• Show a possible approach to scoring and conformance based on discussion to date
• Provide a starting point to make the decisions that need to be made

Disclaimer

• I have included wording for guidelines, functional outcomes, scoring levels and tests to act as examples for the purpose of clarity.
• I have placed a stake in the ground for percentages and other cutoffs. They are currently based on a 90% success. They will need revision based on more testing and discussion.
• Whenever possible I pulled text from existing standards and subgroup work.
• When I could not find existing language or language was undecided I made a series of decisions based on testing and my best educated guess.
• Wording choices and these decisions are not final. The first step is to note where the discussion is needed.

2

Documentation Hierarchy for Conformance

• Functional Need Categories
• Example: Visual
• Guidelines
• Example: Visual Alternatives
• Functional Outcomes
• Groups tests outside of conformance
• Example: Provides text alternative for non-text content
• Tests
• 1 Atomic - Tests whose requirements can be consistently applied across systems (within the same type of tech). Comparable to WCAG 2.x A and AA tests.
• Some but not all of these tests can be automated.
• Example: Image has non-empty accessible name
• Example: Image accessible name is descriptive
• 2A Contextual - Tests that are more difficult to meet, require additional expertise and knowledge of subject (plain language), or can only be tested within the context of the system being assessed (affordances)
• Example: Alternative text uses plain language
• 2B Holistic- Tests that involve the entire path, typical usability and AT testing
• Example: Screen reader user understands the non-text content within the context of the task

3

See Document Hierarchy Section in History/Working Notes for more details

Scope Related Definitions

• Path - A single view or the complete series of views and the specific components & content needed to complete a task from end-to-end.
• A path defines both the views and components needed for conformance
• A component without its associated view is not considered a path
• While some tests may require data outside the path (consistent help location) conformance is only reported against the path
• A path can be an entire set of views that make up a site
• View - All content visually and programatically available without an interaction equivalent to loading a new page or state
• Views include pages, states (in single page applications), and comparable units within web related technology.
• Note: This needs more work. In this approach, I have treated content such as drop downs and error messages to be within a View, but content that mimics a page reload to be a seperate view even if its on a “single page” app. If we can find an existing term and definition for this concept we should use that.

​

4

See Scope Section in History/Working Notes for more details

Scope: Four Ways Conformance

Conformance is defined for paths. Minimal conformance is defined for content, components, or views.

• Path (Both % and failures on the path apply, Bronze-Gold conformance possible)
• Conformance defined for paths. However, a conformance claim may be made to cover one path, a series of paths, or multiple related paths.
• All content and components included within the views on the stated path are in scope
• View (Only % apply, Minimal conformance only)
• Conformance defined for views. However, a conformance claim may be made to cover one view or a representative sampling of views.
• All content and components in view are in scope.
• Content (Only % apply, Only content related methods apply, Minimal conformance only)
• Conformance defined for a defined amount of content.
• Component (Only % apply, Only component related methods apply, Minimal conformance only)
• Conformance defined for a single component or set of components.

5

See Scope Section in History/Working Notes for more details

Scoring Process

1. Identify the task and associated path or paths to be tested
2. Identify the content and views needed to complete the path
3. Run all 1 Atomic tests for all views within a path including components
4. Score each test
5. Write down the # of failures on content needed to complete the path
6. Indicate when a test is not applicable
7. Calculate the Atomic results
• Normalize results
8. Run 2A Contextual and/or 2B Holistic tests
• If 100% on Atomic tests OR
• 90% or greater and no failures on the path
9. Score Contextual and Holistic tests
10. Calculate final conformance scores

​

Note: When testing and improving accessibility, all tests should be run. When Atomic errors are fixed, Contextual and Holistic successes apply

6

See Scoring Section in History/Working Notes for more details

Calculate Final Conformance Scores

• Assign an adjectival rating to each functional Need Categories and overall
• Inaccessible: Less than 90% or any failures in path
• Minimally Conformant:
• 90-99% atomic tests and no failures in path for path based conformance
• 100% atomic tests for view, component, and content based conformance
• Substantially Conformant: 90% of Atomic tests and 90% either Contextual or Holistic tests (Path)
• Bronze: 100% Atomic tests (Path)
• Silver: Bronze and 90% either Contextual or Holistic tests (Path)
• Gold: Bronze and 90% of both Contextual and Holistic tests (Path)
• Conformance requires 100% of atomic tests in all functional categories and overall

7

Questions to Answer With Testing

1. Will 1, 2 or 3 levels of normalization work best?
2. Does adjectival or percentage based normalization work best?
3. Which tests should allow an absence of something to count as a positive?
• Absence of flashing is a pass in 2.x
• Should absence of a timeout be a pass in 3.x?
• Document how this change affects scores
4. Where should the cutoff points be for minimally conformant, substantially conformant, Bronze, Silver and Gold be?
• What are useful and realistic cut off points (% pass) that both ensure users with disabilities can succeed and companies can meet the requirements.
5. Explore and clarify the relationship between failures in path and errors in related views.
• Seeing 100% but 1 failure in path makes it hard to understand
• Equal confusion seeing 45% errors on view and 0 failures in path
6. What is the best way to organize functional categories?
7. Should contextual test results map to guidelines as well or only a general average?

8

Questions to Answer Through Discussion

1. Better word choice and/or definitions for the terms Path and View
• Clarify the similarities and differences between path and process
• Better define path, failures on path, and errors within the larger view and how they affect scoring
2. Better word choice and/or definitions for testing types
3. Choose Scope Option 1, Option 2, or something else
4. Decide how to write holistic tests
5. Confirm functional needs
6. Decide on how to write functional outcomes
7. Finalize proposed Guidelines
8. Finalize proposed Functional Categories
9. What is the best way to measure usability test results across a path in % (error rate, overall completion rate, etc)?
10. How are states handled within views and components?
• Can a component have a state? For example: can a component made up of a <label> and <input> have a "default" state (whatever is presented to the user on View load) and an error state?
• Can a View have a state? For example: a page with and without a modal dialog displayed?
11. Clarify the difference between content and component.
• In 2.x Components are part of content. In this they are not
• Maybe just remove the difference and reference only content
12. Discuss the legal ramifications of “essential” and including mental health
13. Better clarify the difference between a snapshot of conformance and ongoing accessibility testing
14. Should certain tests always be on path?
15. Is there a way to better address the cumulative effect beyond the benefit they get from reporting out on functional categories

9

Test Snapshots

10

Some Errors Off Path

11

• In WCAG 2.1 this would fail 6 SC
• This is minimally conformant because it has 90% or more overall and in all categories and no failures on path

Many Errors Off Path

12

• In WCAG 2.1 this would fail 10 SC
• This is inaccessible because Visual, Memory, and Executive are below 90%.

Flashing Off Path

13

• In WCAG 2.1 this would fail 1 SC
• This is inaccessible because essential is below 90%

One Failure In Path

14

• In WCAG 2.1 this would fail 1 SC
• This is inaccessible because of the failure in path.

Standalone Login Form - View

15

• In WCAG 2.1 this would fail 12 SC
• This is inaccessible because it must hit 100% for the view (not path based)

Silver Rating

16

History/Working Notes

Additional details, notes, and examples follow for reference

17

Requirements

18

Requirements (1/2)

• Multiple Ways to Measure: Tests and procedures must generate results that can be verified. In addition to pass/fail success criteria, WCAG 3.0 must include other ways of measuring success such as adjectival rating, rubrics, percent completion, task-completion, and user research with people with disabilities. This expanded approach to measurement must allow the standards to address 1) needs of people with disabilities and 2) types and scales of technology that are difficult to address in the 2.x conformance model.
• Flexible Maintenance and Extensibility: WCAG 3.0 must provide a maintenance and extensibility model that can be more easily updated to better meet the needs of people with disabilities using emerging technologies and interactions.
• Multiple Ways to Display: WCAG 3.0 must be made available in different accessible and usable formats so the guidance can be customized by and for different audiences.
• Technology Neutral: WCAG 3.0 must be expressed in generic terms to apply to more than one platform or technology. Technology-neutral wording provides the opportunity to apply the core guidance to current and emerging technology, even if specific technical advice doesn't yet exist.

19

Requirements (2/2)

• Readability/Usability: The core guidance of WCAG 3.0 must be understandable by a non-technical audience. Text and presentation must be usable and understandable through the use of plain language, structure, and design.
• Regulatory Environment: WCAG 3.0 must provide structure, methodology, and content that facilitates adoption into law, regulation, or policy. It must clearly state intent and make the purpose and goals transparent in order to assist when questions or controversy occur.
• Motivation: WCAG 3.0 must motivate organizations to go beyond minimal accessibility requirements by providing a scoring system that rewards organizations which demonstrate a greater effort to improve accessibility.
• Scope: WCAG 3.0 must provide guidance for people and organizations that produce digital assets and technology of varying size and complexity. This diverse group of stakeholders includes content creators, browsers, authoring tools, assistive technologies, and others.

20

Scope

21

Scope Related Definitions

• Path - A single view or the complete series of views and the specific components & content needed to complete a task from end-to-end.
• A path defines both the views and components needed for conformance
• A component without its associated view is not considered a path
• While some tests may require data outside the path (consistent help location) conformance is only reported against the path
• A path can be an entire set of views that make up a site
• View - All content visually and programatically available without an interaction equivalent to loading a new page or state
• Views include pages, states (in single page applications), and comparable units within web related technology.
• Note: This needs more work. We want content such as drop downs to be in scope but content that mimics a page reload to be out of scope even if its on a “single page” app. If we can find an existing term and definition for this concept we should use that.

​

22

Scope Option 1: Path Centered Conformance

23

Conformance is defined for paths. However, a conformance claim may be made to cover one path, a series of paths, or multiple related paths.

• Paths may represent a single view, a part of the entire site/application/other, or the entire site/application/other

Assumptions

• The scope of a conformance claim must be stated
• Conformance only applies to the stated scope
• More scoping approaches exist, but this and the next option currently best fit within this proposal
• Organizations’ compliance testing will leverage W3’s conformance model but not necessarily follow the exact breakdown

Scope Option 2: Four Ways Conformance

Conformance is defined for paths. Minimal conformance is defined for content, components, or views.

• Path (Both % and failures on the path apply, Bronze-Gold conformance possible)
• Conformance defined for paths. However, a conformance claim may be made to cover one path, a series of paths, or multiple related paths.
• All content and components included within the views on the stated path are in scope
• View (Only % apply, Minimal conformance only)
• Conformance defined for views. However, a conformance claim may be made to cover one view or a representative sampling of views.
• All content and components in view are in scope.
• Content (Only % apply, Only content related methods apply, Minimal conformance only)
• Conformance defined for a defined amount of content.
• Component (Only % apply, Only component related methods apply, Minimal conformance only)
• Conformance defined for a single component or set of components.

24

Scope: Issues/Decisions Needed

1. Better word choice or definitions for the terms Path and View
2. Choose Scope Option 1, Option 2, or something else

25

Document Hierarchy

26

Documentation Hierarchy for Conformance

• Functional Need Categories
• Example: Visual OR Use Without Vision
• Guidelines
• Example: Visual Alternatives
• Functional Outcomes
• Either part of conformance or helps to group tests outside of conformance
• Example: Provides text alternative for non-text content
• Tests
• 1. Atomic
• Example: Image has non-empty accessible name
• 2A Contextual
• Example: Alternative text uses plain language
• 2B Holistic
• Example: Screen reader user understands the non-text content within the context of the task

27

2.1 and 3.0 Documentation

28

Structure - 2 Level Normalization

29

1

Many

Structure - 3 Level Normalization

30

How Tos

Methods

2. Guidelines

0. Tests

1. Functional Outcomes

3. Functional Categories

Functional Needs

Subsets

1

Many

General Guidance

Structure - Relational Diagram

31

Key Points Based on Hierarchy (1/2)

• Tests
• Tests are the most basic level of conformance
• Write them to be as granular as possible
• Example: Text has minimum contrast; meta viewport does not prevent zoom
• Tests are technology centered
• Functional Needs
• Most basic level of user need
• A statement that describes a specific gap in one’s ability, or a specific mismatch between ability and the designed environment or context. (approved 17 July)
• Examples: Use without color perception, Use with limited vision
• Functional Outcomes
• Functional Outcomes group tests by Functional Needs
• A functional outcome will include a set of tests and a set of functional needs
• Contrast related tests affect both Use without color perception and Use with limited vision
• The set of functional needs within a functional outcome must be consistent across tests
• Zoom related tests affect primarily Use with limited vision so these need to be in separate functional outcome from color and contrast related tests
• The opposite is true for color related tests

32

Key Points Based on Hierarchy (2/2)

• Guidelines
• Guidelines group tests by functional outcome or group functional outcomes (depending on the number of conformance levels)
• A guideline will include a set of related functional outcomes
• Guidelines may affect multiple functional need categories
• Guidelines must be technology neutral
• Functional Categories
• Functional categories are the top level of normalization
• Functional categories group functional needs and the choices on how to group them may vary between guidelines
• Regardless of grouping, functional categories can be expressed at one of two levels
• Higher level: Visual, Auditory, Motor, Cognitive, etc
• Lower Level: Without Vision, With Limited Vision, Without Perception of Color, Memory, etc.
• The higher the level the functional categories are set at, the fewer guidelines are needed
• Balance the need for a short, understandable list of guidelines with representing the differences in accessibility requirements between different functional needs

​

33

Tests

• 1 Atomic
• Example: Image has non-empty accessible name
• Unit of Measure: Image
• Scoring Method: Percentage (# images with accessible names/# images in page/state)
• Scope: Full, Content, Component
• 2A Contextual
• Example: Alternative text uses plain language
• Unit of Measure: Alternative text block
• Scoring Method: Adjectival
• 2B Holistic
• Example: Screen reader user understands images within the context of the task
• Unit of Measure: Path
• Scoring Method: Pass/Fail

34

Functional Needs (1/2)

1. Use without harm or risk
2. Use without vision
3. Use with limited vision
4. Use without color perception
5. Use with limited color perception
6. Use with limited depth perception ·XR·
7. Use with limited orientation or spatial tracking ·XR·
8. Use with photosensitivity (too much or too little)
9. Use without hearing
10. Use with limited hearing
11. Use with limited auditory processing (speech)
12. Use with (age and Presbycusis related) sensorineural hearing loss Use without vision and hearing
13. Use without vision and hearing

​

14. Use with vestibular issues
15. Use without spatial auditory awareness or perception ·XR·
16. Use without mobility
17. Use with limited mobility ·XR·
18. Use with limited reach or range ·XR
19. Use without hands
20. Use without multiple touchpoint gesture
21. Use with limited strength
22. Use without fine point control
23. Use without physical tracking speed
24. Use with tremors
25. Use with limited tactile perception, sensory processing, or touch pressure sensitivity
26. Use with chronic pain impacting input or interaction modality

35

Functional Needs (2/2)

27. Use without vocalization
28. Use with limited vocalization or volume
29. Use with limited ability to focus attention
30. Use with limited ability to direct attention
31. Use with limited ability to shift attention
32. Use without ability to read
33. Use with limited ability to recognize written language
34. Use with limited ability to comprehend written language
35. Use without ability to write
36. Use with limited ability to correctly write (or type) words and use punctuation
37. Use without understanding symbols
38. Use without understanding metaphors, idioms, euphemisms, or specific dialect of culture or location
39. Use with limited ability of math and numeric concepts

​

40. Use with limited compositional skill
41. Use with limited coordinational skill
42. Use with limited short-term or working memory
43. Use with limited medium or long-term memory
44. Use with limited sensory memory
45. Use with limited planning, organization, sequencing, and execution ability
46. Use with limited emotional control and self monitoring
47. Use with limited judgement
48. Use with debilitating fear or anxiety
49. Use with interocular transfer of visual memory
50. Use with limited phonological or phonemic awareness
51. Use with autonomy or agency
52. Use without privacy

​

36

Functional Outcomes

• Functional Outcome (Draft Definition): A statement that describes a singular objective of a user has been met – usually in the context of a task or overall goal – that may need to name or cite a functional need.
• Note
• A functional outcome must map to a set of tests and a set of functional needs
• The set of functional needs within a functional outcome must be consistent across tests
• Examples to Date
• Organizes text into logical chunks to make locating information easier and faster.
• Uses visually distinct headings so sighted readers can determine the structure.
• Provides semantic structure to support assistive technology-- headings are coded as headings.
• Conveys a sense of hierarchy that helps the user explore and navigate the text material.
• Content aids understanding and reading comprehension; avoids causing confusion and relying on memory
• Users perceived the headings
• Users have visually perceived headings
• The visual presentation of has sufficient visual contrast
• Non-text content can be changed into other forms people need, such as large print, braille, speech, symbols or simpler language
• The structure can be visually and programmatically determined

37

Guidelines

• Adaptable
• Auditory Alternative
• Customizable
• Distinguishable
• Navigable
• Predictable
• Prevent Errors
• Reduce Auditory Distractions
• Reduce Visual Distractions
• Safe
• Structured
• Understandable
• Visible Navigation
• Visual Alternative

38

Functional Categories

Functional Needs Based

• Essential
• Includes mental health
• Visual
• Auditory
• Speech
• Motor/Mobility
• Attention
• Language & Literacy
• Learning
• Memory
• Executive

Functional Performance Criteria Based

• Essential
• Without vision
• Limited vision
• Without perception of color
• Without hearing
• With limited hearing
• Without speech
• Limited manipulation
• Limited reach and strength
• Limited language, cognitive, and learning abilities

39

How Tos/Methods

40

Document Hierarchy: Issues/Decisions Needed

• Confirm functional needs
• Decide on how to write functional outcomes
• Determine Guidelines
• Determine Functional Categories
• Decide how to write holistic tests
• Determine Guidelines

41

Scoring

42

Test Scoring

All Scoring methods normalize to percentages

• Percentage: # correct/# instances
• Good for tests where units of measure (# links, # images) are clear
• Pass/Fail: 100%/0%
• Good for tests with a simple yes no answer (language of page)
• Adjectival
• Good for content centered tests such as reflow where units of measure are unclear
• Each test would need to decide on how many options are appropriate and defined each option with clear examples
• Bad (0%), OK (50%), Good (100%)
• None (0%), Some (25%), Half (50%), Most (75%), All (100%)
• Terrible (0%), Bad (20%), Needs Work (40%), OK (60%), Good (80%), Great (100%)

​

43

Normalization

• Normalization is used to balance out different size tests across disability types
• This proposal normalizes using averaging
• Every level of normalization reduces the impact of a single test result
• 2 levels
• Average all tests for a Guideline
• Average all the Guidelines for a Functional Need Categories
• 3 levels
1. Average all tests for a Guideline
2. Average all Guidelines for a Functional Outcome
3. Average all Functional Outcomes for a Functional Need Categories

44

Scoring Process

• Identify the task and associated path or paths to be tested
• Identify the components and pages/states needed to complete the path
• Run all 1 Atomic tests for all pages/states within a path including components
• Score each test
• Write down the # of failures on components needed to complete the path
• If a test is not applicable, select “Not Present”
• Calculate 1 Atomic results
• Normalize results
• Run 2A Contextual and/or 2B Holistic tests
• If 100% on Atomic tests OR
• 90% and no failures on components needed to complete the path
• Score Contextual and Holistic tests
• Calculate final conformance scores

​

Note: When testing and improving accessibility, all tests should be run. When Atomic errors are fixed, Contextual and Holistic successes apply

45

Calculate Final Conformance Scores (Option 1)

• Assign an adjectival rating to each functional Need Categories and overall
• Inaccessible: Less than 90%
• Minimally Conformant: 90-99% atomic tests and no failures in components within the path for path based conformance
• Substantially Conformant: 90% of Atomic tests and 90% either Contextual or Holistic tests
• Bronze: 100% Atomic tests
• Silver: Bronze and 90% either Contextual or Holistic tests
• Gold: Bronze and 90% of both Contextual and Holistic tests
• Conformance would require at least Bronze in all functional categories and overall

46

Calculate Final Conformance Scores (Option 2)

• Assign an adjectival rating to each functional Need Categories and overall
• Inaccessible: Less than 90%
• Minimally Conformant:
• 90-99% atomic tests and no failures in components within the path for path based conformance
• 100% atomic tests for view, component, and content based conformance
• Substantially Conformant: 90% of Atomic tests and 90% either Contextual or Holistic tests (Path)
• Bronze: 100% Atomic tests (Path)
• Silver: Bronze and 90% either Contextual or Holistic tests (Path)
• Gold: Bronze and 90% of both Contextual and Holistic tests (Path)
• Conformance would require at least Bronze in all functional categories and overall

47

Why lower levels must map 1:many upwards

Using current WCAG 2.1Text Alternatives as an example

• 1.1.1 Non Text Content
• Image accessible name is descriptive Without Vision, Limited Vision, Limited Cognitive
• Image has non-empty accessible name Without Vision, Limited Vision, Limited Cognitive
• Image not in the accessibility tree is decorative Without Vision, Limited Vision, Limited Cognitive
• Time-based media has description Without Hearing, Limited Hearing, Limited Cognitive
• Time-based media description is descriptive Without Hearing, Limited Hearing, Limited Cognitive
• If we do not split these based on the grouping of functional categories, than failures that hurt visual disabilities will show up as failures against auditory disabilities and vice versa.
• To maintain validity - each level needs to be grouped in a way that keeps the set of functional categories consistent at the test level
• 1.1.1 Visual Non Text Content
• Image accessible name is descriptive Without Vision, Limited Vision, Limited Cognitive
• Image has non-empty accessible name Without Vision, Limited Vision, Limited Cognitive
• Image not in the accessibility tree is decorative Without Vision, Limited Vision, Limited Cognitive
• 1.1.1 Auditory Non Text Content
• Time-based media has description Without Hearing, Limited Hearing, Limited Cognitive
• Time-based media description is descriptive Without Hearing, Limited Hearing, Limited Cognitive

48

Normative vs. Non-normative

• Functional Categories (Normative)*
• Functional Needs (Non-Normative)
• Guidelines (Normative)
• How Tos
• Methods (Non-normative)
• Functional Outcomes (Normative)
• Tests (Non-normative)
• 1 Atomic**
• 2A Contextual
• 2B Holistic

​

49

Scoring: Issues/Decisions Needed

• Should we normalize at 2 or 3 levels?
• Confirm normative/non-normative decisions
• Should functional categories be normative?
• Should atomic tests be normative?
• What are the best cutoff points. I’ve used 100% but a lower score might be better. See speaker notes

50

Reporting

51

Key Points

• Reporting includes a single total and a breakdown by functional category
• Conformance would require at least Bronze in all functional categories and overall

52

Path Based Final Scoring % - Functional Needs

53

Path Based Final Scoring % - Functional Performance

54

Path Based Final Scoring % - Functional Needs

55

Path Based Final Scoring % - Adjectival

56

Reporting: Issues/Decisions Needed

• Adjectival or Percentage based reporting?
• What is the best way to indicate the ratings in each functional category across all three types of tests?

​

57

Resources

• Silver Taskforce Main Page
• Accessibility Metrics Report
• Mapping of WCAG 2.0 to Functional Performance Criteria
• WCAG to Silver Outline Map
• Path Example
• NIST CIFT - Measuring/Reporting Usability Tests

58