Five Problems with Comprehensive Website Accessibility Audits

For some time now, the consensus was that the best way to make your website or application conform to accessibility standards is to proceed with a comprehensive, in-depth audit of the various templates and interfaces that constitute your property.

However, as time has passed, the realization has set in that comprehensive audits may not always be the best way to begin your accessibility conformance journey. Of course, every situation is different, and every company has a different idea of what they're willing to do to conform to accessibility standards. In some cases, a comprehensive audit might make sense. But for many clients, an agile approach may make the most sense, with the most reasonable entry from a budget perspective.

And this post, we'll look at some of the issues that may arise when conducting site-wide or app-wide testing and make recommendations about how an alternative approach could be beneficial.

Problem 1: They Are Time Consuming

This is one of the most significant problems with comprehensive accessibility audits. They take a lot of time. This is best illustrated by walking through a hypothetical scenario. Let's say you have a web application with twenty different interfaces or templates. You procure the services of a testing firm to provide a findings report and make recommendations so that you can remediate the entire property. This process takes a while to complete, and it will take quite some time before you have any indication about your site's performance relative to the standards.

In the time it takes for an auditing firm to deliver a comprehensive report about the entire project, remediation could have already been in progress. A better approach would be to test small application segments and present those issues first rather than wait for the entire report to be completed. This means that you can be actively producing tangible results while the testing is still happening, and it also introduces efficiencies which I'll cover in our next point.

A quick turnaround is an immediate protection against legal demand letters that may come your way. It shows a fierce commitment to conforming with concrete steps already in motion.

Problem 2: Issues Repeat Themselves. Often.

Audits traditionally have one thing in common: repeatedly finding the same problems. This can be a bit painful to produce, leading to bloated findings reports and increased testing costs simply because the same issue is being documented repeatedly. As I mentioned earlier, testing in segments makes a lot of sense, and this is one area where a smaller sample can be beneficial in the long run. Because almost every website or application is template based these days, once developers understand the basics of an issue, they should be able to apply those changes site-wide. This means that testing of subsequent interfaces should be much more efficient. As this process continues and more interfaces are tested, the findings presented should be smaller, reducing your testing costs.

Of course, not many testing agencies would prefer to tell you this because it decreases billable time. But this is simply too logical a concept to ignore. Conducting testing in a more agile approach, piece by piece benefits everyone.

Problem 3: They Are One Dimensional

Accessibility is one area in the tech world with a lot of nuances. Sadly, many testing organizations are trying to quickly scale their operations, trying the same approach with all different types of clients, hoping to rapidly complete audits and move on to the next customer. This leads to reports that claim to be comprehensive but are missing a key element: an interpretation of how the standards can be thoughtfully applied to your specific scenario.

Too many testing and auditing companies are simply offshoring their testing efforts, and the results are often underwhelming. There simply is a limited connection between the people performing the test and the owners and operators requesting them. Accessibility is and always will be an issue surrounding people, and those issues are difficult to scale. 

Even worse are the firms that conduct testing with only one toolset. It doesn't get more one-dimensional than that. A proper comprehensive accessibility conformance strategy involves testing in multiple environments, with numerous tools, and with domestic testing personnel who are interpreting not just technical implementation but also the context of interfaces. The nature of one-time audits is that the testing firm receives a paragraph or two of background information and then presents the test results. We find this to be entirely too limiting.

Problem 4: Developers often have no idea how to interpret results

Comprehensive audits are typically delivered in gigantic reports overwhelming to the folks who need to remediate the code base. The stress of working with a giant report is compounded by the fact that so many developers have a limited understanding of what accessibility standards are and what the necessary fixes are to comply. Even worse, this is compounded by the fact that most testing organizations aren’t employing developers to guide the remediation.

Rather than delivering a giant report, we found that clients and their developers perform better when presented with a sample of templates or interfaces, including the issues found and the recommended remediation methods. This allows developers to quickly address problems and show progress to management. Overall, it builds positive momentum. Developers can focus on sprints, which is a much more logical workflow for them in the first place. As mentioned earlier, much of the work from initial sprints is applicable across a code base, making future remediation even more efficient. All of this leads to better results quicker.

Also, it's easier to educate developers as they proceed with the remediation of a smaller snippet of interfaces than if they know that there is a giant findings report they're going to have to shuffle through.

Problem 5: The Audit is Done. Now What?

The worst part about a comprehensive audit is the nature of the engagement with the testing firm. Audits are a project which are priced according to a specification. That means there's a beginning, a middle, and an end. After the project concludes, many testing firms don't have a solution for being an ongoing accessibility partner. They have no mechanism to hop on a call for ten minutes to discuss an issue.

This means that even if you've proceeded with remediation, you will be responsible for ongoing monitoring, maintenance, and continued compliance with the standards yourself, as well. This can be challenging for organizations that don't have an in-house accessibility presence. A comprehensive audit is a one-time engagement that leaves you with a capabilities gap instead of a partner who can take on this work for you.

A Better Approach

At NP group, we believe that a better methodology for digital accessibility conformance is a holistic approach that allows for agility, iteration, and interaction between the testing firm and the client. Accessibility is and always will be a human issue that requires a human approach to correcting. It starts with an organization's cultural adaptation and acceptance of the need to comply with the standards. Compliance is a process that includes multiple rounds of testing, remediation, retesting, and, finally, documentation to support making a conformity claim. But moreover, there's a need for ongoing education for everyone who works within a digital property. This means developers, designers, and content managers. A one-time engagement with a testing firm is not a strategy. It could be a completely wrong thing to do.

When Should You Consider a Comprehensive Audit?

As I said at the beginning of the post, there are some circumstances where a comprehensive audit may make sense. The most compelling argument for a complete audit is when a client is reasonably assured that their site will perform well in the audit and requires documentation (such as a VPAT) that would indicate the same. It means more to have a third-party review and document an organization's conformance with the standards rather than an organization conducting that testing themselves.

This means that if your company is seeking to do work with the federal government, as an example, or institutions of higher education, it may make sense to have an audit conducted and have the findings documented so that these materials are available during the sales cycle or procurement process of your products and services.

Of course, this also means that the bulk of remediation was already conducted, and there's already an understanding within the organization of the importance of accessibility, plus a strategy in place. In this sense, using an agency or testing firm to conduct a full-service audit is for confirmation more than anything else.

Wrapping Up

In wrapping up this post, I want to discuss how we've reached the above conclusions. As we evolved our accessibility consulting business, we noticed many of the problems above crop up throughout the process. We also noticed that clients were uncomfortable with the high price point that an audit often resulted in, which would derail their accessibility efforts.

The last thing we want to see is that our clients ignore the issues of accessibility because they can't figure out a way to devise and implement a strategic approach to conformance.

Because of this, we introduced our “Accessibility Ops” program, a retained services-based approach to working with clients wishing to conform to accessibility standards. Please visit our Accessibility Ops page For more information about how our approach works and whether your organization could benefit.