SBA’s Equations Were Reasonable and Estimated Default, and Recovery Rates Were in Line with Historical Experience

We found that the econometric equations that SBA used to estimate  defaults, prepayments, and recoveries were reasonable, although other  equations could also be reasonable. In addition, SBA did not include any  economic variables in its equation for estimating recoveries. Current guidance is either silent or unclear  about supporting documentation needed to explain the development of  econometric models used to generate credit subsidy estimates for the  budget and financial statements. Moreover, as a practical matter, this documentation would help facilitate  SBA’s and other agencies’ annual financial statement audits. Documenting the basis for selecting and rejecting variables from an  econometric model used to develop credit subsidy estimates is an  important internal control that would also help to provide financial  statement auditors reasonable assurance that a bias was not introduced  into the credit subsidy estimates by systematically excluding variables to  influence the subsidy rate in a particular direction. Although at the time of our review, some errors  in its data existed in SBA’s databases, the nature and magnitude of these  errors was unlikely to significantly alter the subsidy rate. SBA Had a Process to Identify and Correct Data Errors

The primary method that SBA used to help ensure the integrity of its loan  data is its Form 1502 reconciliation process. However, from  an audit perspective, SBA’s lack of adequate documentation of the model  development process precluded us from (1) independently evaluating the  model’s development; (2) determining whether SBA used a sound and  consistently applied method to select and reject variables to be included in  the model; and (3) determining whether a bias in selecting variables existed  in the model. Without adequate  documentation, SBA will be unable to transparently demonstrate the  rationale and basis for key aspects of models that provide important cost  information for budgets, financial statements, and congressional decision  makers. Key contributors to this report are listed in  appendix V.

Objectives, Scope, and Methodology

As agreed with your staff, we (1) assessed the reasonableness of the  model’s econometric equations and evaluated the model’s estimated  default, prepayment, and recovery rates based on the 7(a) program’s recent  historical loan experience; (2) identified additional steps the SBA could  take to further enhance the reliability of its subsidy estimate produced by  the model; (3) reviewed SBA’s process for developing the subsidy model;  (4) evaluated the model’s supporting documentation, including its  discussion of what variables were tested and rejected; and (5) determined  what steps SBA has taken to ensure the integrity of the data used in the  model and determined whether these data are consistent with information  in its databases. Assessing the Reasonableness of the Model’s Econometric Equations and Evaluating the Model’s Estimated Default, Prepayment, and Recovery Rates

To analyze the model, we obtained from SBA copies of the model as  approved by the Office of Management and Budget (OMB), along with the  loan-level data that were used to develop the subsidy estimates. 8. 6.
Why GAO Did This Study

The Small Business Administration (SBA) approved about $8.6 billion in loan guarantees through its 7(a) loan program in fiscal year 2003. SBA must estimate the subsidy cost of this program. Since fiscal year 2003, SBA has been using econometric modeling to estimate the subsidy. This report reviews SBA's estimation methodology and equations, assesses the default and recovery rates the model produced, identifies ways to enhance the estimates' reliability, describes the process for developing the model, and analyzes SBA's data.

What GAO Found

From an economics perspective, SBA's econometric equations were reasonable, and its model produced estimated default and recovery rates that were in line with historical experience. However, from an audit perspective, SBA's lack of documentation of the model development process precluded GAO, and others, from independently evaluating the model's development and determining if SBA used a sound and consistently applied method to select and reject model variables. Taking into account economic reasoning and research, SBA's econometric equations for estimating defaults, prepayments, and recoveries were reasonable. SBA's equations used a limited set of variables; equations using other variables could also be reasonable but would produce different estimates. Since an estimate is an approximation, no one estimate can be considered accurate, and reasonable estimates can fall within a range of values. The model's estimated default and recovery rates were in line with recent historical experience. SBA could improve its estimation methodology by periodically checking for and correcting errors and should consider adding more borrower information, such as credit scores. Some errors in the model resulted in understating the estimated program costs. SBA used the expertise of other agencies and a contractor to develop its model and worked closely with the Office of Management and Budget (OMB), which must approve the methodology agencies use to estimate subsidies. OMB officially approved the model in the fall of 2002. SBA did not adequately document its model development process, including alternative variables considered and rejected, to enable external reviewers to assess the process that was used. Further, GAO and two other independent reviewers could not determine whether a bias existed in the model by systematically excluding variables to influence the subsidy rate in a particular direction. Adequate documentation, a key internal control, would enable SBA and other agencies to demonstrate the rationale and basis for key aspects of the model that provide important cost information for budgets, financial statements, and congressional decision makers and facilitate SBA's annual financial statement audit. Current OMB and other guidance is either silent or unclear about the level of documentation necessary for credit subsidy model development. SBA had a process to help ensure data integrity and data consistency in the equations with the loan-level data in its databases. Although errors existed in SBA's data systems, the magnitude and nature of these errors were not likely to significantly affect the subsidy rate.