Automated vs Manual Accessibility Testing

Marisha Lamont-Manfre, PhD

Mike Williamson

Addie Hugen

bit.ly/a11y-testing-summer

Introduction

Marisha

• Accessibility & Usability Assessment Manager since end of 2021.
• BS in mathematics, PhD in Research Methodology

Mike Williamson, ATP

• Assistant Director/Deputy Digital Accessibility Officer, University of Colorado Boulder
• Adjunct Faculty, CSUN
• Specialization in Low vision/blindness, adult populations, and digital accessibility
• RESNA ATP (Assistive Technology Professional)

Learning Objectives

• Understand that automated testing alone does not provide a complete picture
• Understand why digital accessibility is about user experience
• Learn best practices and resources to help implement digital accessibility into workflow

​

We're Not All at the Same Place...�And That's Okay

• Learning/understanding is a process
• You may be at different places than others
• Resources differ
• Not all processes will look the same

Questions to the audience:

What is Accessibility?�What is Usability?

​

What is accessibility and usability?

• We define accessibility as that all people having access to the same information, interactions, and services as everyone else.
• Usability is when the technology is: 
• Functional
• Convenient
• Consistent
• Comfortable
• We care about both. 
• Accessibility standards can be met while the product is still unusable or difficult to use. 

​

How can we measure accessibility?

• There are many ways to measure, assess, and explain accessibility. The two you may have heard of are: 
• Web Content Accessibility Guidelines (WCAG) developed by the World Wide Web Consortium (W3C)
• VPAT (Voluntary Product Accessibility Templates) that vendors will often create and provide with their product
• To fill out a VPAT or to check for WCAG compliance, someone may conduct an accessibility test on a product. While there are different ways of testing, two common approaches are automated and manual testing. 

​

Digital Accessibility in Higher Education

• Digital accessibility is found in all areas of life, but below is a list of examples of where we should consider digital accessibility in higher education:
• Video content
• 3^rd party ed-tech tools
• Websites
• Library databases
• Learning management modules
• Procurement
• OIT software

​

Digital Accessibility Testing

• Accessibility testing is "a subset of usability testing where the users under consideration have disabilities that affect how they use the web." (W3C, 2019). 
• Usability testing evaluates "how usable a product is for people with disability" (Deque, 2014)
• The goal is for accessible and usable sites and software that all users can utilize. ​

​

Automated Testing

Automated testing relies on digital tools that "enable you to audit thousands of documents, web pages, or even multiple websites simultaneously." (Siteimprove, 2022)

​

Automated Testing Pros & Cons

Pros:

• Requires less time and resources (Siteimprove, 2022) 
• Can be free or low cost (essential Accessibility, 2018)
• Results are immediate (essential Accessibility, 2018)
• Quantify how accessible a product is

​

Cons: 

• There can be false positives or false negatives (essential Accessibility, 2018)
• Gives yes/no ratings and not available to provide nuance of issues like alternative text. 
• Reports indicate low accuracy rates 

​

Manual Testing

• Manual accessibility testing uses humans to navigate the website or platform using assistive technology to look for errors. 
• Testers note where there are accessibility issues. 
• Issues can be: 
• Areas not screen reader compatible (Siteimprove, 2022)
• Cannot be accessed by keyboard
• Navigation or readability issues
• Lack of alternative text or labels

​

Manual Testing Pros & Cons

Pros:

• More nuanced
• Allows for true AT users (or people with disabilities) who have lived experiences to assess content

​

Cons: 

• Time – manual testing takes time to go through each page
• Cost – there are testers that need to be paid for their time

​

Mental Maps

• How someone makes sense of their digital environment:
• Document Structure
• Naming conventions (calling something what it is)
• Contextual queues
• Consistent operations

​

How Automated Testing Falls Short

Limitations

• Automated testing alone cannot catch all accessibility or usability issues.
• Studies have shown that automated testing tools only "find 20 to 30% of accessibility issues" (Bunch, 2021). 
• Demo

​

Screen Reader Demo

Using: Two examples of the same PDF (good and bad)

Key takeaways: what you see is not always what you get

​

Manual Testing at CU Boulder

The way that we test has worked for our office and program. While this may not be the best course for other institutions, there are lessons we want to share. 

​

CU Manually Tests

CU Boulder's Assessment & Usability team manually tests digital products. We have developed a methodology that works for our office. 

1. Learn about the product
2. Write task list (or testing script)
3. Test the product using the testing script. 
4. Validate testing notes & compile report.
5. Report is reviewed & sent to the customer

​

CU Manually Tests (part 2)

• We test with a focus on usability. Products can be technically accessible by WCAG standards but may not be usable.
• The report uses four categories:
• Severe – the issue breaks the program
• Significant – the issue is difficult to navigate or "violates" WCAG
• Minor – the issue impedes usability with slight difficulties that's easily overcome
• Usability – issue affects all users

​

Screen Readers – Our Foundational Tool

• Why do we use screen readers?
• Screen reader accessibility captures more than just issues that will impact users who are blind or have low vision.
• Since screen readers read the code, issues impacting all users can be found. 
• For example, if a button is not a programmatic button, then that will also disrupt speech-to-text use.

​

Experience of True AT Users

• Importance of true AT users (or users with disabilities)
• The W3C mentions that there are two groups that should conduct testing: experts and users (2019). 
• They note: "No amount of developer inspection and screening can substitute for the raw clash between a user and a website." 
• True AT users can uncover more technical issues and experience the "subtle interactions between web content and assistive technology" (W3C, 2019). 

​

Other Considerations

• Keep these in mind:
• Color Contrast
• We use the Colour Contrast Analyser by TPGi to test for color contrast. 
• Closed Captions
• Captions need to have a 99% accuracy. Auto captions on YouTube and other sites are generally 80% accurate (3Play Media, 2022). 

​

Artificial Intelligence (AI)

• AI is not a replacement for manual accessibility testing.
• Similar issues with subjectivity and understanding the user impact.
• AI powered overlays are also becoming more popular, but there are issues with those and should be avoided.

​

What can you do?

“The power of the Web is in its universality. Access by everyone regardless of disability is an essential aspect.” 

Tim Berners-Lee, director of W3C

​

Nurturing Digital Accessibility Proficiency And Cultural Competence From An Organizational Perspective

• Accessibility is not a “feature”
• Understanding there is nuance (“it depends”)
• Digital Accessibility Policy
• Training
• Talk about it!
• Let others see you doing it!

​

Nurturing Digital Accessibility Proficiency And Cultural Competence From A Human Perspective

• Disability is an important part of the diversity and inclusion conversation
• Awareness of lived experiences and identities
• Buy-in to human condition
• Experiences vary, they are individual

​

Ask Questions!

• Whether working with vendors, the institutions, or using digital tools in the classroom, here are questions you can ask: 
• Vendors:
• How was the product tested?
• If it was just automated, call it out!
• Do they have a VPAT or a Roadmap?
• Do not take VPATs for their word. Ask about roadmaps and if they're willing to work with you to address accessibility issues/concerns

Ask Questions! (Part 2)

• Institution: 
• What are the digital accessibility policies?  
• How is the institution testing for accessibility?  
• Are there processes for reviewing products? 
• Who should be contacted for accessibility questions/support? 

​

• Online Digital Tools:  
• Am I asking students to engage in activities or use tools that will exclude them?
• Do I know how accessible this tool is?
• What accessibility information is online?  
• Who can you contact (vendor or institution) if there is an accessibility issue?

Thank you!

Questions? Email AUL@colorado.edu

Submit Your Questions

Use the link or QR Code to submit questions to the CU Boulder Digital Accessibility Office (DAO)!

​

We will use your questions to guide us through Friday’s Q&A from 10 to 10:50 AM

bit.ly/camp-questions2026