Non-Homeland Security Primary Performance Measures Are Generally Sound and Data Are Generally Reliable, but Weaknesses Exist

While the six non-homeland security primary performance measures are  generally sound, and the data used to calculate these measures are  generally reliable, we found weaknesses with the soundness of three  measures and the reliability of the data used in one measure (see table 2). All six measures cover key program activities and are objective,  measurable, and quantifiable, but three are not completely clear, that is,  they do not consistently provide clear and specific descriptions of the  data, events, or geographic areas they include. The Coast Guard’s  processes for entering and reviewing its own internal data are likely to  produce reliable data. However, processes for reviewing or verifying data  gathered from external sources vary from source to source, and for the  marine environmental protection measure, the processes are insufficient. Measure without criteria or guidance to accurately reflect program  results and ensure objectivity. Challenges Exist in Using Measures to Link Resources to Results, but the Coast Guard Is Working on Ways to Address Them

While the primary measures for the Coast Guard’s six non-homeland  security programs are generally sound and use reliable data, challenges  exist with using the primary measures to assess the link between  resources expended and results achieved. Each of the Coast Guard’s primary measures for its six non- homeland security programs meets our criteria of covering a key activity. For  example, the primary performance measure for the marine environmental  protection program relates to preventing oil and chemical spills. Developing a system or model that could realistically take all of these  other factors into account is perhaps impossible, and it would be a  mistake to view this second challenge as a need to do so. Coast Guard Has Developed a Range of Initiatives to Forge Better Links between Resources and Results

The Coast Guard is actively seeking to address such challenges, as those  discussed above, through efforts, some of which have been under way for  several years. Recommendations for Executive Action

To improve the quality of program performance reporting and to more  efficiently and effectively assess progress toward achieving the goals or  objectives stated in agency plans, we recommend that the Secretary of  Homeland Security direct the Commandant of the Coast Guard to:    Refine certain Coast Guard primary and secondary performance  further clarifying the ice operations primary measure by clearly  and consistently describing the geographic area and number of  waterways included in the measure; the living marine resources  primary measure by clearly and consistently reporting the scope  of the measure; and the search and rescue primary measure and  the search and rescue “percent of lives saved after Coast Guard  notification” secondary measure by reporting those incidents or  data that are not included in the measures;    developing measurable performance targets to facilitate  assessments of whether program and agency goals and  objectives are being achieved for the 11 living marine resources  secondary measures and the 1 marine environmental protection  secondary measure, “Tokyo and Paris memorandums of  understanding port state control reports,” that lack annual  targets; and    establishing agencywide criteria or guidance to help ensure the  objectivity and consistency of the search and rescue program’s  “percent of property saved” secondary performance measure.
Why GAO Did This Study

Using performance measures, the Coast Guard explains how well its programs are performing. To do so, it reports one "primary" measure for each program (such as percent of mariners rescued) and maintains data on other, "secondary" measures (such as percent of property saved). Concerns have been raised about whether measures for non-homeland security programs accurately reflect performance, that is, they did not rise or fall as resources were added or reduced. For the six non-homeland security programs, GAO used established criteria to assess the soundness of the primary measures--that is, whether measures cover key activities; are clearly stated; and are objective, measurable, and quantifiable--and the reliability of data used to calculate them. GAO also used these criteria to assess the soundness of 23 selected secondary measures. Finally, through interviews and report review, GAO assessed challenges in using measures to link resources to results.

What GAO Found

While some opportunities for improvement exist, the primary measures for the Coast Guard's six non-homeland security programs are generally sound, and the data used to calculate them are generally reliable. All six measures cover key program activities and are objective, measurable, and quantifiable, but three are not completely clear--that is, they do not consistently provide clear and specific descriptions of the data, events, or geographic areas they include. Also, the processes used to enter and review the Coast Guard's own internal data are likely to produce reliable data; however, neither the Department of Homeland Security (DHS) nor the Coast Guard have policies or procedures for reviewing or verifying data from external sources, such as other federal agencies. Currently, the review processes vary from source to source, and for the primary measure covering marine environmental protection (which concerns oil and chemical spills), the processes are insufficient. Of the 23 secondary performance measures GAO assessed, 9 are generally sound, with weaknesses existing in the remaining 14. These weaknesses include (1) a lack of measurable performance targets, (2) a lack of agencywide criteria or guidance to ensure objectivity, and (3) unclear descriptions of the measures. Two main challenges exist with using primary measures to link resources to results. In one case, the challenge is comprehensiveness--that is, although each primary measure captures a major segment of program activity, no one measure captures all program activities and thereby accounts for all program resources. The other challenge involves external factors, some of which are outside the Coast Guard's control, that affect performance. For example, weather conditions can affect the amount of ice that must be cleared, the number of aids to navigation that need repair, or mariners that must be rescued. As a result, linking resources and results is difficult, and although the Coast Guard has a range of ongoing initiatives to do so, it is still too early to assess the agency's ability to successfully provide this link.