"Progress towards conformance"

May 2026

Initial idea

Show progress on the way to conformance, a score.

The score could involve:

• the % of provisions met
• the % of a site that meets
• a combination
• Or anything else that would be a meaningful measure of progress
• Divided by disability type?

Can be used both to compare two products for accessibility or to plot progress on accessibility of a product/site over time.

Still working on this aspect.

Known issues with scoring based on WCAG 2

• If you have a large amount of pages, a simple “which SCs have you failed across everything” is not very useful.
• There could be 1 issue on 1 page, or 100s per page.
• Counting instances can create distortions (and was not popular in FPWD)
• If you don’t take into account the number of possible fails, an absolute number would penalise larger pages.
• Having 1000s of minor or repeating issues makes it looks harder to fix than it is.
• No count of issues or score can account for context-based impact
• E.g. a missing name on a delete button is much worse than missing name on a link to something the user doesn’t care about.
• Scoring by functional need (is good) but will have some hard questions to answer
• Which types - How many?
• What if a provisions helps 5 disabilities - but affects some more than others. Does it count for all? Equally?

Known issues with scoring based on WCAG 2

• A simple “which SCs have you failed across everything” is not very useful.
• This becomes even more problematic as you increase the number of pages
• There could be 1 issue on 1 page, or 100s per page.
• Counting instances can create distortions (and was not popular in First Public Working Draft)
• If you don’t take into account the number of possible fails (denominator), an absolute number would penalise larger pages.
• Having 1000s of minor or repeating issues makes it looks harder to fix than it is. -- but it is a more accurate picture of the pain of users in trying to use the page/site
• No count of issues or score can account for context-based impact
• E.g. a inaccessible name on a submit or delete button is much worse than inaccessible name on a link to something the user doesn’t care about.
• Scoring by functional need (is good) but will have some hard questions to answer
• Which types of functional need - How many (just 4 major CHVP+other) or 10 or 15?
• What if a provisions helps 5 disabilities - but affects some more than others. Does it count for all? Equally?

Previous Resources

• 2020 Scoring example
• Option 1 from CSUN 2023
• Option 2 from CSUN 2023
• Metrics and plan for evaluating conformance scoring

Goal of this work

Define metrics which quantify the distance from conformance, either levels or full conformance.

The group considered:

• What metrics to experiment with
• Gathering test cases with real (or at least realistic) test results
• Compare the results to known usability-testing, and/or picking known good/poor examples.
• Report back to the AGWG

Goal of this work

Define metrics which quantify the distance from conformance, either just conformance at base level or between each level as well.

The group considered:

• What metrics to experiment with
• Gathering test cases with real (or at least realistic) test results
• Compare the results to known usability-testing, and/or picking known good/poor examples.
• Report back to the AGWG

Metric exploration

• Number of core requirements failed per page/view
• Could/should these score more than supplemental?
• Number of supplemental requirements failed per page/view
• How many pages/views the issue is present on out of the total pages
• Instances per-page, providing a percentage that averages out
• Also having scoring for functional-needs or categories
• Instances of flashing (and other non-interference provisions), impact on score

Other considerations

Does it work better to present it as positive or negative number?�E.g. “Barrier score” (lower better), or “Conformance score” (higher better). Will start with higher as better.

Metric exploration

• Number of core requirements failed per page/view
• Could/should these score more than supplemental?
• Number of supplemental requirements failed per page/view
• How many pages/views the issue is present on out of the total pages
• Instances per-page, providing a percentage that averages out
• Also having scoring for functional-needs or categories
• Instances of flashing (and other non-interference provisions), impact on score
• Weighted scoring across 4 + “other” disability categories

Other considerations

Does it work better to present it as positive or negative number?�E.g. “Barrier score” (lower better), or “Conformance score” (higher better). Will start with higher as better.

Metrics not pursued

• Remediation complexity
• Even “simple” things like a change of color may have to go through multiple teams / layers of approval.
• Could be a side score?�
• Provisions with not tested/no data (because we were working with WCAG2 data from previous testing and the provision in question (and thus evaluation data) did not exist for this exercise)
• We are largely working from WCAG2 test data and mapping it to a subset of WCAG3

Metrics not pursued

• Remediation complexity
• Even “simple” things like a change of color may have to go through multiple teams / layers of approval.
• Could be a side score?
• This appeared to be more of an “effort” score than a measure of accessibility/conformance.�
• Provisions with not tested/no data
• We are largely working from WCAG2 test data and mapping it to a subset of WCAG3

Plan for the “spike” group

• Created tabs for each product, up to two per person
• Each rows has provision short name; assertions are excluded
• Each page/view has a column
• Each cell is completed with pass / fail / NA / no data
• Results are recorded in the spreadsheet, mapping from WCAG 2 to 3
• Note a fail as “1”, or the number of instances if counting. 0 = pass or NA
• The average score is calculated using:
• Average the number of fails across conformance units (pages/views)
• Create an overall percentage for the scope of testing.
• Giacomo will also look at gathering results from some WCAG2 cases, and use instances for fails.

Plan for the “spike” group

• Created tabs for each product, up to two per person
• Each rows has provision short name; assertions are excluded
• Each page/view has a column
• Each cell is completed with pass / fail / NA / no data
• Results are recorded in the spreadsheet, mapping from WCAG 2 to 3
• Note a fail as “1”, or the number of instances if counting. 0 = pass or NA
• The average score is calculated using:
• Average the number of fails across conformance units (pages/views)
• Create an overall percentage for the scope of testing.
• Gregg created a DATA page that included all provisions with impact on 4+1 disability groups. (He took a first pass at weights and benefits but invites experts for each disability group to review and change scores and add benefits for each for their group)
• Giacomo will also look at gathering results from some WCAG2 cases, and use instances for fails.
• Giacomo also took a pass at scoring with weights on DATA page (that will auto-adjust as weights on data page evolve)

Potential gaming areas identified

• Including only a subset of pages with the least issues
• Will need to be addressed in WCAG-EM & Policy documents
• Passing several supplemental requirements to cancel out core failures
• Could be mitigated by “gates” (pass all core before next level), or increasing weight for core provisions.

Potential gaming areas identified

• Including only a subset of pages with the least issues
• Will need to be addressed in WCAG-EM & Policy documents
• Problem doesn’t arise if any valid sampling is done (valid sampling does not allow this)
• Passing several supplemental requirements to cancel out core failures
• Could be mitigated by “gates” (pass all core before next level), or increasing weight for core provisions.
• Also since supplemental are not required (at least at base level) they would not affect the score (at least until they are required).

Case studies / examples

Case 1: A Large content site ( measuring instances)

• Have data for WCAG2 instances (fails & total possible instances) for a subset of WCAG 3 provisions
• Generally good experience (people with visual disabilities), usability testing score 85%
• Averaged 71% “progress to conformance”, not including instances.
• Averaged 87% including total possible instances and subjective severity.
• By weighted category: Visual 69%, Cog 74%, Hearing 87%, Physical 82%

Case 2: An e-commerce site (measuring instances)

• Have data for WCAG2 instances (fails and total possible instances), sub-set of WCAG 3 provisions
• Generally poor experience (people with visual disabilities), usability testing score 60%
• Averaged 73% “progress to conformance” not including instances.
• Averaged 80% including total possible instances and subjective severity.
• By weighted category: Visual 68%, Cog 73%, Hearing 91%, Physical 82%

Conclusions from instance testing

• Including proportional instances (where you count how many times it could have failed) does improve the score somewhat. �I.e. gets it correlates more closely to the user-experience.
• Due to the number of “Not applicables”, most sites score above 50% by default, even pretty awful ones.
• The working area is really the 60-95% range.
• Weighting by disability category didn’t appear to add a lot, but it would be useful to try this on more sites. (but the goal of disability category is to have separate scores for each category - not to create a unified score)

Case 3: A Gov app (no instance counting)

• Generally poor user experience.
• App, so relatively small-scope of interface (lots of not-applicables)
• Data is pass/fail per page, not instances.
• Average pass rate: 86%
• Weighted score (core = 2 x supplemental) across all pages: �66% (based on all provisions failed)

Case 4: A good website (no instance counting)

• Generally good user experience.
• Website
• Data is pass/fail per page, not instances.
• Average pass rate: 96%
• Weighted score (core = 2 x supplemental) across all pages: �88% (based on all provisions failed)

Conclusions on the non-instance scoring

• Average pass rate seems to reduce differences between products
• Totalling the issues and weighting core provisions increased the differentiation.
• This type of scoring would give the appearance of dragging-down generally good sites with some poor areas.

“Barrier score” compared to WCAG fails

The “Barrier score” is internal to Gain (formerly Nomensa), and provides a subjective score based on WCAG fails and impact of instances. It is a subjective estimation by the auditor. A lower score is better.

Taking the first 37 audits in a random list, a mix of large and small websites/apps, the barrier score was correlated with:

• The number of WCAG issues found (i.e. any number of alt-text issues is one SC 1.1.1 fail).
• The number of issues reported (which separates types of issues. E.g. content and decorative issues are separately reported).

Barrier score correlation

The correlation of WCAG 2 fails to barrier score was 0.80 (out of 1), showing a “strong positive correlation”. (sheet) Less so for reported issues (0.68).

Case study of internal scoring from a large org

• A large organisation providing internal training for testing hundreds of sites.
• Rating scale 1-5 or NA
• Rating includes severity based on frequency and judgement
• Approach was usability tested and refined
• Each checkpoint defined what a 1, 3 and 5 was and allowed for 2 and 4 when people couldn’t decide.
• Some checkpoints (e.g. language tag) were either 1 or 5
• Checkpoints were mapped against key functional needs (blind, low vision, color blind, motor, deaf/hard of hearing, cognitive, seizures)

Lessons learned

• Judgement call on severity was the hardest part of training though results were fairly consistent
• Because of this, the system doesn’t work well for conformance. Better for internal monitoring.
• A site/app that had never thought about accessibility but was built with a modern framework came in around 4.3 (86% the scale)
• The result, visible in the table is that the difference between inaccessible (4.3) and perfect (5) is only 14% of the scale
• Moving between a 4.8 and a 5 was the largest amount of time/effort (likely because harder work was put towards the end)
• Usually (except when something was built without responsive design) by the time a team reached 4.8 all functional needs were supported except blind and had motor impairments

Wrapping up

Score improvement factors

When trying to align a score with the user-experience for people with disabilities, the following factors appear to improve it:

1. Severity: Assessing each instance for severity. High-severity instances have a large impact on the score. Subjective, may not be possible to include.
2. Proportionality: For each provision (per page/view), what proportion of instances fail? E.g. 100 images on a page, 50 fail, that provision scores 50%.
3. Provision level: Such as core & supplemental, but more levels could be added. A safety level (e.g. Flashing) should be included for scoring.
4. Provision weight: Like having multiple levels, each provision is ranked by it’s average impact (e.g. functional images matter more than decorative images).

Score improvement factors

When trying to align a score with the user-experience for people with disabilities, the following factors appear to improve it:

• Severity: Assessing each instance for severity. High-severity instances have a large impact on the score. Subjective, may not be possible to include.
• Proportionality: For each provision (per page/view), what proportion of instances fail? E.g. 100 images on a page, 50 fail, that provision scores 50%.
• Provision level: Such as core & supplemental, but more levels could be added. A safety level (e.g. Flashing) should be included for scoring.
• Disability Categories: If disability categories were used they would not be used to create a total score but rather be separate to ensure that the fails did not fall disproportionately on one group.

Other conclusions

• If we can’t take into account “severity” due to it being contextual, then it must be framed as “progress to conformance”, not an assessment of usable-accessibility for the user.
• We assumed not-applicable scores the same as pass, removing “NA” from the scoring would be a big change. It would probably penalise simpler sites.
• We need to prevent cascading failures, e.g. if there are 4 audio description (AD) items, should you fail the conditional ones? Adding an AD that fails other requirements could score worse than one without AD.
• Once a site considers accessibility at all, it tends to get above 50%. Due to many “NAs”, that means that the useful scoring is often from 70-95%

Other conclusions

• If we can’t take into account “severity” due to it being contextual (and perhaps even if we could?), then it must be framed as “progress to conformance”, not an assessment of usable-accessibility for the user.
• We assumed not-applicable scores the same as pass. Removing “Pass it you don't have it” from the scoring would be a big change. It would probably penalise simpler sites.
• We need to prevent cascading failures, e.g. if there are 4 audio description (AD) items, should you fail the conditional ones or just the main one of having AD? Conclusion- it should fail main and all other related provisions otherwise - adding an AD that fails other requirements could score worse than one without AD. (This does however make that provision have greater weight in the score than if the sub requirements were just part of the main requirement. Do it right or it doesn’t count.)
• Once a site considers accessibility at all, it tends to get above 50%. Due to many “Pass if you don't have it” or “Pass if you have nothing this applies to” , that means that the useful scoring is often from 70-95%

Potential reporting requirements

A score will depend on various factors, especially which pages/views are chosen so when providing a score based on an optional conformance claim:

• Which pages/views were included.
• (Optional?) Media count (e.g. number of videos)
• (Optional?) Component count

​

These would provide some context for the score because certain provisions are triggered when certain content is present.

Potential reporting requirements

If valid sampling is not “required” - (non-valid sampling is allowed) then a score could depend on various factors, like which pages/views are chosen when providing a score based on an optional conformance claim:

• Which pages/views were included.
• (Optional?) Media count (e.g. number of videos)
• (Optional?) Component count

​

These would provide some context for the score because certain provisions are triggered when certain content is present.

Next steps

Assuming the group wishes to continue:

• Run the scoring over a larger set of sites
• Check correlation between usability testing results and the different scoring mechanisms.
• See if other companies can automate �
• Draft a version of this for the editor’s draft of WCAG3.

Next steps

Assuming the group wishes to continue:

• Run the scoring over a larger set of sites
• Check correlation between usability testing results and the different scoring mechanisms.
• See if other companies can automate
• Pick a model (or best of each approach) and run them and compare
• Try scoring in 5 categories CDPVS safety) with and without weighting
• Draft a version of this for a document to accompany the editor’s draft of WCAG3
• (It would not be part of WCAG3 directly because it is not a requirement and is not conformance. �It is a note about something that would be useful for comparing non-conforming products and for tracking progress).