Conformance Conversations

2024

Note: If you are new to this conversation, start by reviewing the references and vocabulary

Topics to Cover

• Guideline structure (In Process)
• How are outcomes, methods and tests organized?
• Are overall conformance levels determined? If so, are they assigned at the outcome, method or test levels?
• What content is normative?
• Where do user needs fit in?
• Are outcomes, methods, and tests additive (AND) or comparable (OR)?
• Where do assertions fit in?
• Outcome verification
• Are additional results available other than pass/fail at the item/view/user process/product level?
• Do different outcomes have different scopes - how do we build that in?
• Are levels built into the outcome level or below?
• Overall conformance
• How should scope be defined ?
• How do we build in tolerance?
• Do we include the idea of critical errors?
• Evaluating models
• Process
• Evaluation Criteria

Guideline Structure

Example

Summary of where we are

• Conformance Proposal Comparison
• Outcomes trending towards additive (AND)
• Methods trending towards methods as comparative (OR between)
• Test are all additive (AND)

Conformance

Conformance is Not Compliance

• Conformance relates to standards - W3C and AG remit
• Compliance relates to laws, regulations, and policy - not W3C and AG remit
• Exclusions and reasonable efforts concepts are more relevant to Compliance

​

Proposed Structure

• Overview
• Guidelines are very high level
• Outcomes have an AND between them
• Required Methods have an OR between them
• Tests would have AND
• Assertions are under an Outcome, inside or next to the decision tree.
• Best Practice Methods (Optional), also have an AND between them
• Some Methods may have both a Required and Best Practice level within them.
• User Needs are under Outcomes
• Example

Baseline

Baseline (Set of Required Outcomes)

• Compliance does not equal the baseline
• We can use different criteria than WCAG 2 to set the baseline
• The number of outcomes in a baseline can fall on a continuum

Baseline scenarios - The extremes

Everything is in the baseline - All requirements are included�

• A lot of testing required�
• Nothing is ‘extra’, or going above and beyond typical effort�
• Expensive and unrealistic - not adopted as a standard

No baseline - Evaluator selects what to do from preset list

• “Accessible” could result in items that no-one with a disability could use.�
• Could use a score, with a number of outcomes required to pass (or number you are allowed to fail).

A baseline can be based on…

• WCAG 2 - User-need filtered by testability & feasibility�
• Automatability - Includes things which can be tested automatically.�
• Functional needs cross-section - X number of requirements per functional need.�
• Content type - If you include something (e.g. forms, multimedia), then include a minimum set of requirements for each thing.�
• Adaptability - Within outcomes, could optimise for adaptability with fall-back authoring methods.

Anything else you can think of?

Possible Overall Conformance Models

With Baseline

• Levels
• Bronze: Requirements (Baseline)
• Silver: Requirements + x%
• Gold: Requirements +x+y%
• Baseline + x%
• Baseline + badges

Without Baseline

• x% of guidelines/outcomes
• Accumulating points based on guidelines/outcomes
• (Or scoring barriers instead)

​

Discussion Questions

• How important is a baseline?
• Every outcome is important to someone so if we have a baseline, how do we prioritize and justify?

Answer this question

With a scoring approach, if a site scored enough to conform but still had a (keyboard) inaccessible main navigation, is this acceptable?

​

(Examples could also be accessible authentication, plain language etc.)

Reasoning: Any “buffer” e.g. scoring will allow for this gamification.

Baseline is not everything

• Baseline < Regulatory - The baseline could include less than the intended regulatory level. E.g. We recommend that Silver is the level rather than baseline.�
• Assertions - Assertions could be included in the ‘baseline’.

Baseline hybrid solution

Baseline + extras / badges

• Include everything at bronze in the baseline.
• Silver / Gold levels could then be scoring based.

Non-baseline scenarios…

Bronze level then building

• Score > 90% of Bronze requirements
• Score > 50% of Silver requirements
• Score > 50% of Gold requirements

Scoring by functional category

• Score over 90% within each functional category for bronze level requirements
• Badges for passing sets of requirements at Silver and Gold.

Resolution

Proposal agreed in the meeting:

We progress with a baseline and make effort to make it even across functional needs. Prioritize requirements for safety, blockers and adaptation / author support for user-agent features (within outcomes).

Then another level (or two) which is scored, providing a means of progressing.

Applying levels - Baseline

What would go into the baseline level

Things we agreed should be part of the baseline:

• Safety concerns
• Breaking user-agents (UAs) ability to adapt content (e.g. adjust color)
• Author support for UAs (e.g. alt-text)
• Is a blocker (not ‘could be’ a blocker), and cannot be worked-around with user-agent support.

Note - we will have authoring methods / fallbacks for UA methods, there needs to be “accessibility support” for the UA method to be relied on.

Possible structure for levels

The structure we are demo-ing is what we’ll be using for sub-groups to work on.

Is there anything to refine before being used as a template for new guidelines?

Open question about the decision tree: Start with the baseline or start with the higher-level items?

Guidance examples

Example - Adjust color

Requirements:

• Not blocking user-agent adjustments - Baseline
• Minimum text-contrast for ‘locked’ items (e.g. Canvas, SVG, images of text) - Baseline
• Minimum contrast for authored text - Enhancement
• Meaningful information in graphics and interfaces does not rely solely on color to convey meaning - Baseline?

​

Example - Keyboard only

• All functionality can be performed through the keyboard interface only - Baseline
• No keyboard trap - Baseline
• Keyboard interface interactions are consistent - Enhancement
• Action Completion Time - Enhancement
• Comparable Keyboard Usage Effort - Enhancement

Example - Section labels

• Sections of content have clear visual and programmatic labeled - Baseline
• Content is organized with related information is grouped together with the structure presented visually and semantically - Baseline
• Navigation elements are clearly differentiated from content, both visually and programmatically - Baseline for programmatically
• Multiple Ways - Enhancement

Terminology

We have used bronze/silver/gold as placeholders for conformance levels.

We have previously also used bronze/silver/gold for outcome levels.

• Should we maintain that, or switch terms?
• In the examples we have “Baseline” and “Enhanced”.
• Other suggestions: Optimised, Recommended, Best Practice, Required

Bronze may or may not equal the baseline/required outcomes

• Example: Meeting Bronze may be completing all of the baseline outcomes OR Meeting Bronze may be completing all Baseline outcomes plus x% of Enhanced Outcomes
• As we explore conformance options, is it confusing to use the same terms?

​

�

​

Example

• Bronze = Baseline + 50% of enhanced requirements
• Silver = Baseline + 70% of enhanced requirements
• Gold = Baseline + 90% of enhanced requirements

​

Alternative wording:

• Bronze = Base/prerequisites + 50% of enhanced requirements
• Silver = Base/prerequisites + 70% of enhanced requirements
• Gold = Base/prerequisites + 90% of enhanced requirements

​

Alternative model

• Prerequisites/quick check (Complete before proceeding)
• Levels

Goals for 24 June 2024

Goals for discussion

• Identify a few models to try out over the next few months
• Clarify vocabulary
• Initial rules for what goes in the baseline

Goals for model (to be discussed later)

• How does model address issue of pass/fail issue?
• How does model encourage going beyond basics?
• See Requirements Document

Terminology

• Prerequisite (smaller subset)
• “Thing required as a prior condition for something else”
• Include: Safety Issues, Needed for AT to work, Prevents task completion even with ideal AT support
• Baseline (larger subset) -
• “Minimum or starting point used for comparison”
• Include: Prerequisite plus author provided outcomes that aren’t currently met by AT, applicable to all sites and products
• Enhanced
• Outcomes that may be domain specific or can be met other ways

How levels might work - using prerequisites

1. Prerequisite plus % based levels

Level 1: Prerequisites (You can not continue unless all are met)

Level 2: 50% of Enhanced Outcomes

Level 3: 70% of Enhanced Outcomes

Level 4: 90% of Enhanced Outcomes

​

(2) Levels with prerequisite and %

Level 1: All Prerequisites + 50% of Enhanced Outcomes

Level 2: All Prerequisites + 70% of Enhanced Outcomes

Level 3: All Prerequisites + 90% of Enhanced Outcomes

% are placeholders

How levels might work - using baseline

(3) Baseline plus % based levels

Level 1: Baseline

Level 2: Baseline + 50% of Enhanced Outcomes

Level 3: Baseline + 90% of Enhanced Outcomes

​

(4) Set levels only

Level 1: Baseline

Level 2: Baseline + Enhanced

*This is what we have now

Proposal for consolidation with draft categorization

Alternate

(5) Baseline, another level, plus % based of enhanced

Level 1: Baseline

Level 2: Baseline + more

Level 3: Baseline + more + 50% of Enhanced Outcomes

Models to test

Proposed 25th June:

Move forward with (3) and (5) as comparators.

Mark outcomes as prerequisite / baseline / enhanced.

�The “more” would be selected from the enhanced category, criteria TBC.

​

3) Baseline plus % based levels

Level 1: Baseline

Level 2: Baseline + 50% of Enhanced Outcomes

Level 3: Baseline + 90% of Enhanced Outcomes�

5) Baseline, another level, plus % of enhanced

Level 1: Baseline

Level 2: Baseline + more

Level 3: Baseline + more + 50% of Enhanced Outcomes

Outcome template

Outcome template

• Square brackets / yellow indicate hints to delete.�
• Includes all the headings, some examples.�

​

Sub-group process

• Review scratchpad
• Define scope of an outcome
• Gather examples (in scratchpad)
• Construct methods decision tree (in outcome template)
• Fill in methods
• Test with examples
• Refine methods & decision tree
• Apply levels to methods (see next slide)
• Bring back to AG

Completed keyboard focus example

Applying levels to methods

• Prerequisite (small subset)
• Safety Issues, Needed for AT to work, likely to prevent task completion even with ideal AT support.
• Baseline (larger subset) -
• Author provided methods that aren’t currently met by AT, applicable to all sites and products.
• Enhanced
• Can be met in other ways, or may be domain specific.

Examples:

• Image Description Method (Baseline)�
• Decorative Image Methods (Enhanced)�
• Which type of indicator is being used, authored or user-agent? (Baseline)�
• Common words example.

​

Product level conformance

Possibility to consider: Aggregate assertion level

• Separate out assertions and apply them to a level that affects the aggregate
• More ‘maturity’ oriented, providing confidence that it would be passing view-level guidelines
• Examples
• Sample based testing (Similar to WCAG-EM)
• Plain language review
• Usability testing
• Use of design system

Product and view conformance

Two parts to conforming

Product conformance

• Considers the product as a whole

View conformance

• Considers the conformance of a single view

View conformance

A view is conformant if it meets the minimum Outcomes

Views can meet additional Outcomes that would confer a higher level of conformance

Product conformance

Basic product conformance means that either:

• All views within the product meet the minimum level of conformance

OR

• The views within all key user journeys/tasks meet the minimum level of conformance, and
• A random representative sample of views meet the minimum level of conformance

Product conformance, cont

Increased levels of product conformance could include:

• Assertions of best practice activities, such as usability testing, organizational training activities, conformant accessibility statement, etc
• Increased scope or level of view conformance, that is, more views reviewed or higher level of conformance of the views reviewed

Prerequisites

Summary of Conversation to Date (Github Discussion 104)

• There is some support for writing a standard that prioritizes methods that meet outcomes through a combination of platform support and authors avoiding breaking that support.
• This would likely be done by placing the methods that specify authors avoid breaking platform support within the prerequisite level.

Summary of Conversation to Date Cont.

That said, there are concerns that:

• The ability of the author to rely on platform support assumes a platform provides adequate support
• In many cases the platform does not yet provide adequate support
• The platform may provide support but only through a means that is expensive or technically difficult for the user
• When platform support is not adequate, author support must be provided
• Authors need a way to determine when platform support is adequate
• Some situations such as emergency information that should always require author support.

Terminology

• I used "Platform" in the previous slide to mean all support that does not originate from the author, such as the support from the operating system, browser, assistive technology, authoring tools, or a combination of these.
• Is “platform” the correct term? No - conflicts with other definitions
• Alternatives:
• “Stack” and Author
• Stack, Authoring tools, and Author
• Platform, Authoring Tools, and Author
• Platform def? “Everything between the user and the content” or EN301 549
• “User platform”
• Author controlled, platform controlled, user controlled, user controlled via AT

Possible Way to Think About Outcomes in WCAG 3

Guideline:

How would authors determine platform support is adequate?

• Examples for clarification
• Questions of “What is “good enough”?
• Availability
• Does any platform support the requirement?
• How does availability vary by region and language?
• Cost
• Difficulty for user to implement
• Ability to identify if it meets the outcome
• Who determines “good enough”?

TPAC

Slide Deck from TPAC

Minutes

• September 23
• September 24
• September 26 Joint Meetings

Next Steps

• Individual Activity: Participants try out testing WCAG 3
• Chairs: Try flipping % passes and fails
• Group activity:
• Sort outcomes into categories exercise
• Explore alternative conformance models
• Break down guidance into more than 3 levels
• Explore reporting or leveling by functional need
• Document the outcomes where cumulative errors would cause harm or prevent use
• Explore writing considerations for policy makers

Conformance models to explore

1. Required plus %:
• Level 1: Required outcomes;
• All levels above level 1 are based on %
2. Levels: Each of 4-5 level is clearly defined; Each level has a required set of outcomes (preselected to balance functional needs)
3. Hybrid: Level 1: Required outcomes; All levels above level 1 include % AND a subset of required outcomes for that level
4. Required plus functional need %: Level 1: Required outcomes; All levels above level 1 include % AND a minimum % requirement for each functional need�(Pros and cons)

Guideline and Outcome formatting

• GitHub Discussion
• AG Meeting Discussion - agreement to use technical statement format
• Examples:
• Tasks, including login/authentication, can be completed without puzzles, calculations, or other cognitive tests (essential exceptions would apply)
• Accurate names, roles, values, and states are available for interactive components.

Revising existing structure

• Still working the exploratory level
• Finding a need to organize content by a higher grouping
• Agreed at Feb 13th meeting to use technical statements for the more specific guidance level
• We have been referring to those as outcomes but we suggest better aligning the language since the technical format we decided on isn’t an “outcome”
• Suggested way forward:
• Outcomes: Plain language, user-oriented statements of the desired outcome; grouping the guidelines
• Guidelines: Technical statements to meet each outcome

Switching terminology - example

• Outcome: Users can see which element has keyboard focus.
• Present: A visible indicator of the keyboard focus is available.
• Not obscured: The focus indicator is not obscured or partially obscured (more than 50%, TBC)
• Persistent: The focus indicator persists while the element has focus, but does not persist after the element loses focus.
• Distinctive: The keyboard focus indicator uses a style that is distinct from the style of other controls, so that the item in focus can be distinguished without reference to the non-focused state.
• Sufficiently visible: According to the specific method (below), the indicator must be visually discernible whilst navigating.�
• Outcome: Users can understand the meaning of images using assistive technology.
• Images have equivalent text alternatives.

Exercises

• Group 1: Refine outcome and guideline organization
• Group 2: Refine outcomes statements to match agreed upon format
• Group 3: Categorize as Required and Enhanced to help with testing conformance and forming subgroups
• Individual exercise with discussion around meeting

Key points

• We are still at exploratory for this exercise
• We need clarity and consistency but not perfection
• Editors will combine the results

Reference Section

References

• WCAG 3
• Proposals 2018 to Present
• Conformance Proposal Comparison
• Previous Conformance Work
• Use Cases for WCAG 3.0 Conformance
• Subgroup handbook
• Writing testable outcomes

Vocabulary

• Baseline: A core set of required outcomes that must be met before scoring occurs
• Conformance: adhering to technical standards. In this context accessibility standards, such as the W3C Accessibility Guidelines (WCAG).
• Compliance: adhering to laws and other types of policies. In this context accessibility policies, such as the European Accessibility Act (EAA).
• Items: The smallest testable unit. They may be interactive components such as a drop down menu, a link, or a media player. They may also be units of content such as a word, a phrase, a label or error message, an icon, or an image.
• Platform:
• Product: The combination of items, views, and user processes that collectively comprise the web site, set of web pages, web app, etc.
• Programmatically indicated
• User process: A series of user actions, and the distinct interactive views and items that support the actions, where each action is required to complete an activity. A user process may include a subset of items in a view or a group of views.
• Verification: PRocess of establishing the status of a test
• Views: All content visually and programmatically available without a substantive change. Conceptually, views correspond to the definition of a web page as used in WCAG 2, but are not restricted to content meeting that definition. For example, a view could be considered a “screen” in a mobile app or a layer of web content – such as a modal.

Questions for future discussion

• How do we handle outcomes that are not applicable/automatically met either because the platform handles it or the content doesn’t exist?
• How do we define when platform support is “good enough” to allow authors to assume an outcome has been met without author action?
• Can assertions be included in the baseline?
• Two or three levels of conformance?
• Once we have a model that works, how do we simplify it?