SBA officials said they  considered, but did not add, a scorecard element that calculated changes  in the number of small business subcontractors, which the 2016 NDAA  required to be included if data were available. Prime contracting achievement. Subcontracting achievement. The  officials did not provide us a draft plan or details about the plan. Such an evaluation also could assist SBA in understanding whether the  scorecard revisions may contribute to maximizing contract dollars  awarded to small businesses, which is one of the goals in SBA’s strategic  plan. Errors in Published Scorecard Results Weaken Reliability and Perceived Integrity of Scorecard Program

Scorecard results originally published by SBA on May 22, 2018,  contained errors, including one agency scorecard published with an  incorrect letter grade. Agency Officials and Other Stakeholders Expected the Revised Scorecard to Have Little Impact on Small Business Opportunities

Agency officials and representatives of small business groups we spoke  with generally expected the revised scorecard methodology for fiscal year  2017 to have little impact on small business procurement opportunities. Officials from three of the four federal departments and representatives  from the three small business groups we interviewed said they had not  seen any changes in opportunities for small business prime contracting  as a result of the scorecard’s methodological changes. SBA  officials said that SBA does not have a process to ensure that published  scorecard results are accurate. Recommendations

We are making the following two recommendations to SBA:  The SBA Administrator or her designee should complete the design and  implementation of a comprehensive evaluation of the Small Business  Procurement Scorecard aligned with key attributes of effective program  evaluations to assess the effectiveness of the revised scorecard in  measuring agency performance and promoting small business  procurement opportunities. Regarding our recommendation that SBA institute a process to review  scorecards for accuracy prior to publication and a mechanism for publicly  identifying when issued scorecards have been revised, SBA said it had  taken several steps to revise the processes for publishing accurate  scorecard results, including adding steps to compare the prepared  scorecard documents to source documents prior to publication and to  annotate any score corrections that are made to published scorecards. Appendix I: Objectives, Scope, and Methodology

This report describes (1) revisions to the Small Business Procurement  Scorecard (scorecard) methodology for fiscal year 2017 and results of the  fiscal year 2017 scorecard, as well as the extent to which the Small  Business Administration (SBA) plans to evaluate the effects of revisions;  (2) the extent to which SBA’s revised scorecard methodology uses  relevant and reliable information and SBA publishes accurate scorecards;  and (3) views of selected federal agencies and industry stakeholders on  the extent to which SBA’s revised scorecard methodology may encourage  agencies to expand small business procurement opportunities. To examine the changes SBA made to the Small Business Procurement  Scorecard and the rationale for these changes, we reviewed relevant  documents, including the National Defense Authorization Act for Fiscal  Year 2016, SBA’s descriptions of the prior and revised scorecard  methodology, and revised peer review guidance used for the scorecard  element that assesses compliance with section 15(k) of the Small  Business Act. We also compared SBA’s revised scorecard  methodology against the agency’s policies on information quality and  against GAO’s standards for internal control in the federal government.
Why GAO Did This Study

Each year SBA produces a scorecard measuring federal contract spending allocated to small businesses. The 2016 NDAA included a provision for SBA to revise the scorecard's methodology and for GAO to evaluate the effects of those revisions for fiscal year 2017. This report discusses, among other things, (1) SBA's changes to the scorecard methodology and plans, if any, to evaluate the effects of these changes, (2) the extent to which SBA has processes to disseminate reliable information, and (3) views of selected stakeholders on the scorecard's effects on small business procurement opportunities.
GAO analyzed SBA's prior and revised scorecard methodology and results and interviewed officials from SBA, four other federal agencies selected based on small business procurement volume and other attributes, and three groups representing the interests of small businesses.

What GAO Found

For fiscal year 2017, the Small Business Administration (SBA) revised the methodology for its Small Business Procurement Scorecard, which is used to assess federal agencies' progress toward small business procurement goals. SBA made revisions to address requirements specified in the National Defense Authorization Act for Fiscal Year 2016 (2016 NDAA). SBA (1) reduced the share of the total scorecard grade devoted to prime contracting achievement, which is the dollar amount of contracts awarded directly to small businesses, and (2) added an element calculating changes in the number of small businesses receiving prime contracts. SBA made two additional revisions—with input from other agencies' representatives—to increase the share of subcontracting achievement results and peer review of required activities designed to facilitate small business procurement (see figure). In July 2018, officials said they had begun developing a plan to evaluate the effects of the revised scorecard methodology but did not provide a draft plan. Conducting a well-designed and comprehensive evaluation could aid SBA in determining whether the scorecard is an effective tool for helping to achieve the agency's strategic goals.
(Scorecard elements are expressed as a percentage of total scorecard grade.)
The published fiscal year 2017 scorecards originally contained errors, including an incorrect grade and numeric score for one agency, and SBA does not have a process to ensure that scorecard results are published accurately. Although SBA later corrected the errors, the agency did not initially document that scorecards had been changed, which is inconsistent with SBA's policy on information quality. SBA officials said that errors occurred in the process of formatting scorecards for publication. Errors in the published scorecards—and the initial lack of disclosure about corrections—weaken data reliability and may undermine confidence in scorecard data.
Agency officials and representatives of small business groups that GAO interviewed generally expected the scorecard revisions to have little impact on small business procurement opportunities. However, one agency's officials said they would focus more on tracking subcontracting activity as a result of changes to the scorecard.

What GAO Recommends

GAO is recommending that SBA (1) design and implement a comprehensive evaluation to assess scorecard revisions and (2) institute a process for reviewing scorecards for accuracy prior to publication and a mechanism for disclosing corrected information. SBA generally agreed with GAO's recommendations.