NIST manages federal funding of this type of program through its Manufacturing Extension Partnership Program, or MEP. The Advanced Technology Program

In our work on ATP, our objective was to examine, as one way to assess the program’s impact, whether research projects would have been funded by the private sector if they had not received funds from ATP. We found that ATP had funded research projects that would have been funded by the private sector as well as those that would not. The winners were nearly evenly divided when asked if they would have pursued their projects even if they had not received ATP funding. Of the 123 applicants we surveyed, 77, or 63 percent, did not look for funding from other sources before requesting it from ATP. Furthermore, of the 45 applicants that tried to find funding elsewhere before turning to ATP, about half were told by prospective funders that their projects were either too risky or precompetitive—characteristics that fulfill the aims of ATP funding. Most manufacturers responding to our questionnaire—about 73 percent—reported that they believed the programs’ assistance had positively affected their overall business performance. About 15 percent of the respondents reported that they believed the assistance had not affected their overall business performance. Whether the focus is on basic research, applied research, or development, the amount of money spent in that area is taken as an indication of how much research is being performed. We also found that because of the difficulties in identifying the impacts of research, quantitative and qualitative indicators have been developed as proxies to assess R&D results. Quantitative indicators focus mainly on return on investment, patenting rates, and bibliometrics—the study of publication-based data. While varying in the types of indicators they collect, they emphasized the difficulties in measuring R&D’s specific contribution to a company’s overall performance. All of the companies interviewed have increased their expectation that R&D contribute directly to their profitability. Many of the R&D output measures tracked by the private sector do not apply directly to the federal government. Although this is not the only approach that can be taken, this response to the challenges in measuring the impacts of research shows that some progress is being made in response to GPRA.
Why GAO Did This Study

GAO discussed its work related to the National Institute of Standards and Technology's Advanced Technology Program (ATP), Manufacturing Extension Partnership (MEP) Program, and research and development (R&D) performance measures.

What GAO Found

GAO noted: (1) ATP has funded research projects that would have been funded by the private sector as well as those that would not; (2) the award recipients were nearly evenly divided when asked if they would have pursued their projects if they had not received such funding; (3) GAO also found that in most cases, the participants in its survey did not look for funding from other sources, private or public, before trying to obtain funding from ATP; (4) about half of the 45 applicants that tried to find funding elsewhere before turning to ATP were told by prospective funders that their projects were either too risky or "precompetitive", characteristics that fulfill the aims of ATP; (5) manufacturers viewed the manufacturing extension programs' services positively, as was demonstrated in GAO's national survey of manufacturers who had received substantive services from the programs in 1993; (6) most manufacturers responding to GAO's questionnaire, about 73 percent, reported that they believed that the type of assistance they had received from these programs had positively affected their overall business performance; (7) about 15 percent of the respondents reported that they believed the programs' assistance had not affected their overall business performance; (8) the amount of money spent on R&D, the primary indicator of research investment, is useful as an input measure of how much research is being performed; (9) however, the level of spending is not a reliable indicator of research results; (10) GAO found that there is no primary indicator of R&D results; (11) the companies that GAO spoke with collect data on various output indicators, such as return on investment and patents granted, but in general make limited use of them in their investment decisions; (12) instead, they emphasized that R&D contributes directly to the bottom line; (13) because companies are profit oriented, many of the indicators tracked by the private sector cannot be directly applied to the federal government; (14) determining the specific outcomes resulting from federal R&D is a challenge that will not be easily resolved; and (15) however, in response to recent legislation requiring agencies to report on program results, some progress is being made in measuring the impacts of research.