Evaluating the WCAG3 Proposal
Data metrics and resources for researchers and specialists.
Research & Metrics
Research
Metrics and plan for evaluating WCAG3
In 2011, the W3C WAI Research and Development Working Group (RDWG) held an Online symposium on Web Accessibility Metrics. These metrics is this slide deck are based on the Research Report on Web Accessibility Metrics of 2012.
We welcome additional research and recommendations. We need more expert input.
Metrics
Resources for testing special purposes
Example Test Sites
Validity Testing Escape Room site - this a copy of a website that we deliberately modified to have specific flaws that we could test against.
Validity Testing Deque’s “Gefälscht” site - this is a site with errors maintained by Deque.
Example Test Login / Registration Form
Example of a test App
This is an example of testing a mobile app.
Anything else we want to say about it that is notable?
GOV UK’s Accessibility Testing Pages
First Public Working Draft content
The First Public Working Draft has proposed guidelines for:
Testing Tools
Greater detail of tools to test WCAG 3 are on following slides, but for ease of access, they are listed here:
New prototype testing tools
The following slides were developed for the First Public Working Draft. They are outdated. They are included for archival interest.
Testing Clear Words
To test Clear Words, you can use the Silver Writer tool.
Silver Writer / Clear Words Tool
Using The Clear Words Tool
To use the Clear Words tool:
Testing Visual Contrast
Spreadsheet
How to use the scoring spreadsheet
Start with the Tests sheet
Testing Text Alternatives For Images Of Text (HTML)
Testing Clear Words
The Outcomes Sheet
Example Outcomes Sheet
Outcome | Total Passed | Total Critical Failures | Score |
Text alternative available | 84% | 2 | 0 |
Common Clear Words | 0% | 0 | 0 |
Translates speech and non-speech audio | 4.0 | 0 | 4 |
Headings with levels organize content | 100% | 0 | 4 |
Use visually distinct headings | 83% | 0 | 3 |
Convey hierarchy with semantic structure | 51% | 0 | 0 |
The Final Score sheet
Example Final Score Sheet
Functional Categories | Score |
Vision and Visual | 1.3 |
Hearing and Auditory | 4.0 |
Sensory Intersections | 2.0 |
Mobility | 2.0 |
Motor | 2.0 |
Physical and Sensory Intersections | 2.0 |
Speech | 0.0 |
Attention | 1.5 |
Language and Literacy | 2.0 |
Learning | 2.8 |
Memory | 1.5 |
Executive | 3.0 |
Mental Health |
|
Cognitive and Sensory Intersections | 1.8 |
Total | 1.8 |
Critical Failures | 2 |
Feedback
How to apply testing to WCAG3 Metrics?
By testing with known flaws, we can evaluate the metrics of WCAG3 itself.
Some examples of testing we can do are:
Testing the Reliability Metric
Reliability: Do different testers make the same scoring decisions and achieve the same overall score?
Testing the Sensitivity Metric
Test with a set of pages with known accessibility problems. There would be different versions of the set with different levels of accessibility to see how the score changes.
For example, we would have a set where all the problems for screen readers were fixed, another where all the low vision and color contrast problems were fixed. This would help show us if we have successfully balanced the scoring so that all disabilities were treated fairly.
Testing the Adequacy Metric
Have two versions of the same digital product. One would have small changes in accessibility from the other.
We would test both and look to see that the change in score is proportionate to the change in accessibility.
Testing the Complexity Metric
Ask experts to test a site for WCAG2 and compare how long it took to test with WCAG3. Since WCAG3 currently only has 5 guidelines, test WCAG2 only for:
Feedback on Metrics
Please share your data
Questions?
Thanks for your interest.
Francis Storr�francis.storr@intel.com�Github (for W3C work): https://github.com/fstrr
Jeanne Spellman�jspellman@spellmanconsulting.com�Twitter & Github: @jspellman