NIST Has Defined and Implemented an Accreditation Approach That Reflects Relevant Standards but Is Missing Details Needed for Consistent and Verifiable Implementation

NIST’s defined approach to accrediting voting system laboratories largely  reflects applicable HAVA requirements and relevant international  standards, both of which are necessary to an effective program. In particular, because NIST’s  defined program does not, for example, specify the nature and extent of  assessment documentation to generate or retain or specify the version of  the voting system standards to be used, our analysis of NIST’s efforts in  accrediting four laboratories could not confirm that the agency has  consistently followed its defined accreditation program. However, they said that they do not have documented  plans to accomplish this. According to NIST officials, these HAVA requirements are relevant and  important to defining an effective voting system testing laboratory  accreditation program. As a  result, NIST has incorporated key aspects of an effective accreditation  body into its voting system accreditation program. For  example:    One type of checklist (the supplemental handbook checklist) was  prepared for only two of the four laboratory assessments. However, this draft manual omits  important content. Because this manual was not  available for EAC to use on the four laboratory accreditations that it has  completed, the accreditations were performed using a largely  undocumented series of steps. However, they said that they do not yet have  documented plans to accomplish this. With respect to the remaining three HAVA requirements, the draft EAC  accreditation manual also requires (1) that the commissioners vote on the  accreditation of laboratories recommended by NIST for accreditation, (2)  that EAC publish an explanation for the accreditation of any laboratory  not recommended by NIST for accreditation, and (3) that the  commissioners vote on the proposed revocation of a laboratory’s  accreditation. Further, the manual is silent on the steps to be followed and  criteria to be applied in reviewing a laboratory’s application and the  qualifications required for accreditation reviewers. Further, while EAC officials said that they plan to  evolve their approach to VSTL accreditation and to address these gaps,  EAC does not have documented plans for accomplishing this. Available Documentation Does Not Demonstrate EAC’s Basis for Accrediting Laboratories to Date

As of May 2008, EAC has accredited four laboratories, but the  documentation associated with each of these accreditations is not  sufficient to recreate a meaningful understanding of how each evaluation  was performed and how decisions were made, and thus, the bases for each  accreditation were not clear. According to EAC officials, the review steps were not documented. Both NIST and EAC play critical roles in  ensuring that the laboratories that test these two variables have the  capability, experience, and competence necessary to test a voting system  against the relevant standards. To help EAC in evolving its VSTL accreditation program, we recommend  that the Chair of the EAC ensure that the EAC Executive Director  develops and executes plans that specify tasks, milestones, resources, and  performance measures that provide for the following action:    Establish and implement practices for the VSTL accreditation program  consistent with accreditation program management guidance published by  NIST and GAO, including   documentation of specific accreditation steps and criteria to guide  assessors in conducting each laboratory review; transparent requirements for the qualifications of accreditation  reviewers; requirements for the adequate maintenance of records related to the  VSTL accreditation program; and requirements for determining laboratory financial stability. With respect to our finding that NIST’s defined approach for accrediting  VSTLs does not cite explicit qualifications for the persons who conduct  the technical assessments, the institute stated that it does explicitly cite  assessor qualifications for its overall national laboratory accreditation  program, adding that this approach to specifying assessor qualifications  has a proven record of success. As we  state in our report, we reviewed the documentation associated with the  accreditation assessments for these four laboratories, and we found that  all four were not documented in a similar manner, even though they were  based on the same version of the program handbook. Appendix I: Objectives, Scope, and Methodology

Our objectives were to determine whether the National Institute of  Standards and Technology (NIST) and the Election Assistance  Commission (EAC) have defined effective voting system testing laboratory  (VSTL) accreditation approaches, and whether each is following its  defined approach.
Why GAO Did This Study

The 2002 Help America Vote Act (HAVA) created the Election Assistance Commission (EAC) and assigned both it and the National Institute of Standards and Technology (NIST) responsibilities for accrediting laboratories that test voting systems. NIST assesses a laboratory's technical qualifications and makes recommendations to EAC, which makes a final accreditation decision. In view of the continuing concerns about voting systems and the important roles that NIST and EAC play in accrediting the laboratories that test these systems, GAO was asked to determine whether each organization has defined an effective approach for accrediting laboratories that test voting systems and whether each is following its defined approach. To accomplish this, GAO compared NIST and EAC policies, guidelines, and procedures against applicable legislation and guidance, and reviewed both agencies' efforts to implement them.

What GAO Found

NIST has largely defined and implemented an approach for accrediting voting system testing laboratories that incorporates many aspects of an effective program. In particular, its approach addresses relevant HAVA requirements and reflects relevant laboratory accreditation guidance, including standards accepted by the international standards community. However, NIST's defined approach does not, for example, cite explicit qualifications for the persons who conduct accreditation technical assessments, as called for in federal accreditation program guidance. Instead, NIST officials said that they rely on individuals who have prior experience in reviewing such laboratories. Further, even though the EAC requires that laboratory accreditation be based on demonstrated capabilities to test against the latest voting system standards, NIST's defined approach has not always cited these current standards. As a result, two of the four laboratories accredited to date were assessed using assessment tools that were not linked to the latest standards. Moreover, available documentation for the four laboratory assessments was not sufficient to determine how the checklists were applied and how decisions were reached. According to NIST officials, the four laboratories were consistently assessed. Moreover, they said that they intend to evolve NIST's accreditation approach to, for example, clearly provide for sufficient documentation of how accreditation reviews are conducted and decisions are reached. However, they had yet to develop specific plans for accomplishing this. EAC recently developed a draft laboratory accreditation program manual, but this draft manual does not adequately define all aspects of an effective approach, and it was not used in the four laboratory accreditations performed to date. Specifically, while this draft manual addresses relevant HAVA requirements, such as the requirement for the commissioners to vote on the accreditation of any laboratory that NIST recommends for accreditation, it does not include a methodology governing how laboratories are to be evaluated or criteria for granting accreditation. Because the manual was not approved at the time EAC accredited four laboratories, these accreditations were governed by a more broadly defined accreditation review process that was described in correspondence sent to each laboratory and a related document receipt checklist. As a result, these accreditations were based on review steps that were not sufficiently defined to permit them to be executed in a repeatable manner. According to EAC officials, including the official who conducted the accreditation reviews for the four laboratories, using the same person to conduct the reviews ensured that the steps performed on the first laboratory were repeated on the other three. However, given that both the steps and the results were not documented, GAO could not verify this. EAC officials stated that they intend to evolve the program manual over time and apply it to future accreditations and reaccreditations. However, they did not have specific plans for accomplishing this. Further, although EAC very recently approved an initial version of its program manual, this did not occur until after EAC provided comments, and GAO had finalized, this report.