As I wrote to the Secretary of State before the staff report was released, I have serious concerns about the testing process. While it was intended to verify the usability and effectiveness of Vote-PAD for use in an election, the test protocol was planned and implemented by people who have no experience conducting usability testing — particularly with respect to persons with disabilities — and without the oversight of a human factors expert. And, the results were evaluated by people who have no experience in human factors.

My primary concern is that the testing procedures failed to simulate a live election, so it’s difficult to know how the results would compare with the results of using the Vote-PAD in a live election. It is clear, however, that in spite of the artificial environment, and with qualitatively more difficult tasks than voters would perform, participants did very well.

It is notable that, in the staff report on the Vote-PAD, three of the eight pages discussing test results were devoted to write-in issues. However, in the staff report on the Hart eSlate system which was tested a week prior to the Vote-PAD, no write-in issues were raised, even though the consultant’s report points out three severe problems with the write-in feature:

♦ It is possible for the eSlate audio to report that one letter was entered when actually another was entered,

♦ There is no way for a blind voter using the eSlate to review the name of a write-in candidate, and

♦ The eSlate’s audio feedback for the Clear/Backup function is confusing.

Since the eSlate can give inaccurate feedback on write-in names, and — unlike the
Vote-PAD — provides no way for voters to verify the name, it appears that voters with visual impairments can cast a write-in vote on the eSlate system even less reliably than on the Vote-PAD. It’s difficult to understand why the staff didn’t consider the eSlate write-in issues enough of a problem to include in their report.

And because the staff chose not to test the eSlate by having voters with visual impairments actually vote on the system – as they did with Vote-PAD – there is no way of knowing how high the write-in error rate would have been, or how reliably blind voters could have cast write-in votes on those systems. And again, because to our knowledge the Secretary of State has no established standard for objectively measuring such reliability, it is difficult to understand what subjective standard was used to reach the staff’s conclusion.

Certainly write-in candidates present a major challenge for voters using the Vote-PAD, but there is no evidence to suggest that the challenge is greater than it is on other systems, since this stringent testing by people with disabilities has not occurred on any other system certified in California.

In summary, additional testing which accurately depicts a voting situation appears necessary to ascertain the potential benefits or difficulties associated with this product and procedure. Any testing should follow standard usability testing guidelines and research procedures.

As a final point: Assuming a system is accessible, just because it is computerized, is an injustice to the disabilities community.

California owes it to its disabilities community to conduct appropriate accessibility testing on all systems that it certifies for use by people with disabilities. We urge the Secretary of State to conduct an appropriate test of the Vote-PAD for full certification in the near future, and to include appropriate usability testing in its test protocol template for all systems.