Subsequently, the decision was made to conduct testing at Aberdeen Test  Center, which is not NIJ-certified. 3. 4. Army Took Significant Steps during Preliminary Design Model Testing to Run a Controlled Test and Maintain Consistency but Did Not Consistently Follow Established Testing Protocols and, as a Result, Did Not Achieve the Intended Test Objective

		Army Took Significant Steps to Run a Controlled Test and Maintain Consistency

Following are significant steps the Army took to run a controlled test and  maintain consistency throughout Preliminary Design Model testing:    The Army developed testing protocols for the hard-plate (ESAPI and  XSAPI) and flexible-armor (FSAPV-E and FSAPV-X) preliminary  design model categories in 2007. Of potentially greater consequence to the final test results is our  observation of deviations from testing protocols regarding the clay  calibration tests. During  approximately the first one-third of testing, however, Army testers  incorrectly measured deformation at the point of aim, rather than at the  deepest point of depression. The Army  then changed the test plans and modified the contract solicitation to call  for measuring at the point of aim. As a result, at least two  of the eight designs that passed Preliminary Design Model testing and  were awarded contracts would have failed if the deepest point of  depression measurement had been used. During First Article Testing the Army Addressed Some of the Problems Identified in Preliminary Design Model Testing, but Army Testers Did Not Always Follow Established Testing Protocols and Did Not Maintain Some Internal Controls

		During First Article Testing, the Army Addressed Some of the Problems Identified during Preliminary Design Model Testing

PEO Soldier maintained an on-site presence in the test lanes and the Army  technical experts on the Integrated Product Team charged with testing  oversight resolved the following problems during First Article Testing:    The Army adjusted its testing protocols to clarify the required shot  location for the impact test, and Army testers correctly placed these  shots as required by the protocols. While we  did not validate this assertion that rounding was a common industry  practice, one private industry ballistics testing facility said that its practice  was to always round results up, not down, which has the same effect as  not rounding at all. However, the Army did not consistently maintain adequate internal  controls to ensure the integrity and reliability of its test data. For example,  in one case bullet velocity data were lost because the lane Test Director  accidentally pressed the delete button on the keyboard, requiring a test to  be repeated. Army Did Not Formally Document Significant Procedures That Deviated from Established Testing Protocols or Assess the Impact of These Deviations

According to Army officials, decisions to implement those procedures that  deviated from testing protocols were reviewed and approved by  appropriate officials. With respect to internal control issues, Army officials acknowledged that  before our review they were unaware of the specific internal control  problems we identified. Furthermore, one of the remaining  two designs that passed First Article Testing was a design that would have  failed Preliminary Design Model testing if back-face deformation was  measured in accordance with the established protocols for that test. In performing this evaluation, the  independent experts should specifically evaluate the effects of the  following practices observed during First Article Testing:  the rounding of back-face deformation measurements;    not scoring penetrations of material through the plate as a complete  penetration unless broken fibers are observed in the Kevlar backing  behind each plate;  the use of the laser scanner to measure back-face deformations  without a full evaluation of its accuracy as it was actually used during  testing, to include the use of the software modifications and operation  under actual test conditions;  the exposure of the clay backing material to rain and other outside  environmental conditions as well as the effect of high oven  temperatures during storage and conditioning; and  the use of an additional series of clay calibration drops when the first  series of clay calibration drops does not pass required specifications. With respect to the  specific inconsistencies we identified between the test practices and  testing protocols, we recommend that the Secretary of the Army, based on  the results of the independent expert review of the First Article Test  results, take the following actions:    Determine whether those practices that deviated from established  testing protocols during First Article Testing will be continued during  future testing and change the established testing protocols to reflect  those revised practices. DOD also does not  concur with what it perceives as our two overarching conclusions: (1) that  Preliminary Design Model testing did not achieve its intended objective of  determining, as a basis for contract awards, which designs met  performance requirements and (2) that First Article Testing may not have  met its objective of determining whether each of the contracted plate  designs met performance requirements. We did not verify the accuracy  of the Army’s test data and did not provide an expert evaluation of the  results of testing. To understand the practices the Army used and the  established testing protocols we were comparing the practices with, we  met with and/or obtained data from officials from the Department of  Defense (DOD) organizations and the industry experts listed in table 1:  To determine the degree to which the Army followed established testing  protocols during the Preliminary Design Model testing of body armor  designs, we were present and made observations during the entire period  of testing, compared our observations with established testing protocols,  and interviewed numerous DOD and other experts about body armor  testing. After capturing the testing  data in our data collection instruments, we compared our observations of  the way Aberdeen Test Center conducted testing with the testing protocols  that Army officials told us served as the testing standards at the Aberdeen  Test Center. 8. DOD agreed that Army testers deviated from the testing protocols by  measuring back-face deformation at the point of aim.
Why GAO Did This Study

The Army has issued soldiers in Iraq and Afghanistan personal body armor, comprising an outer protective vest and ceramic plate inserts. GAO observed Preliminary Design Model testing of new plate designs, which resulted in the Army's awarding contracts in September 2008 valued at a total of over $8 billion to vendors of the designs that passed that testing. Between November and December 2008, the Army conducted further testing, called First Article Testing, on these designs. GAO is reporting on the degree to which the Army followed its established testing protocols during these two tests. GAO did not provide an expert ballistics evaluation of the results of testing. GAO, using a structured, GAO-developed data collection instrument, observed both tests at the Army's Aberdeen Test Center, analyzed data, and interviewed agency and industry officials to evaluate observed deviations from testing protocols. However, independent ballistics testing expertise is needed to determine the full effect of these deviations.

What GAO Found

During Preliminary Design Model testing the Army took significant steps to run a controlled test and maintain consistency throughout the process, but the Army did not always follow established testing protocols and, as a result, did not achieve its intended test objective of determining as a basis for awarding contracts which designs met performance requirements. In the most consequential of the Army's deviations from testing protocols, the Army testers incorrectly measured the amount of force absorbed by the plate designs by measuring back-face deformation in the clay backing at the point of aim rather than at the deepest point of depression. Army testers recognized the error after completing about a third of the test and then changed the test plan to call for measuring at the point of aim and likewise issued a modification to the contract solicitation. At least two of the eight designs that passed Preliminary Design Model testing and were awarded contracts would have failed if measurements had been made to the deepest point of depression. The deviations from the testing protocols were the result of Aberdeen Test Center's incorrectly interpreting the testing protocols. In all these cases of deviations from the testing protocols, the Aberdeen Test Center's implemented procedures were not reviewed or approved by the Army and Department of Defense officials responsible for approving the testing protocols. After concerns were raised regarding the Preliminary Design Model testing, the decision was made not to field any of the plate designs awarded contracts until after First Article Testing was conducted. During First Article Testing, the Army addressed some of the problems identified during Preliminary Design Model testing, but GAO observed instances in which Army testers did not follow the established testing protocols and did not maintain internal controls over the integrity and reliability of data, raising questions as to whether the Army met its First Article Test objective of determining whether each of the contracted designs met performance requirements. The following are examples of deviations from testing protocols and other issues that GAO observed: (1) The clay backing placed behind the plates during ballistics testing was not always calibrated in accordance with testing protocols and was exposed to rain on one day, potentially impacting test results. (2) Testers improperly rounded down back-face deformation measurements, which is not authorized in the established testing protocols and which resulted in two designs passing First Article Testing that otherwise would have failed. Army officials said rounding is a common practice; however, one private test facility that rounds told GAO that they round up, not down. (3) Testers used a new instrument to measure back-face deformation without adequately certifying that the instrument could function correctly and in conformance with established testing protocols. The impact of this issue on test results is uncertain, but it could call into question the reliability and accuracy of the measurements. (4) Testers deviated from the established testing protocols in one instance by improperly scoring a complete penetration as a partial penetration. As a result, one design passed First Article Testing that would have otherwise failed. With respect to internal control issues, the Army did not consistently maintain adequate internal controls to ensure the integrity and reliability of test data. In one example, during ballistic testing, data were lost, and testing had to be repeated because an official accidentally pressed the delete button and software controls were not in place to protect the integrity of test data. Army officials acknowledged that before GAO's review they were unaware of the specific internal control problems we identified.