Coast Guard Security Risk Models

From 2001 to 2006, the Coast Guard assessed maritime security risk  using the Port Security Risk Assessment Tool (PSRAT), which was  quickly developed and fielded after the terrorist attacks of September 11,  2001. In  addition to providing decision makers with an understanding of how to  interpret any uncertainty in MSRAM’s risk estimates, greater transparency  and documentation could facilitate periodic peer reviews of the model—a  best practice in risk management. MSRAM’s risk assessment methodology also generally aligns with the  NIPP criteria for a reproducible risk assessment. Training. The Coast Guard has  acknowledged these challenges and limitations and has actions underway  to address them and make MSRAM more complete and reproducible. Additional Documentation Could Align MSRAM with NIPP Criteria for Documented and Defensible Risk Assessments

MSRAM is generally documented and defensible, but the Coast Guard  could improve its documentation of the model’s assumptions and other  sources of uncertainty, such as subjective judgments made by Coast  Guard analysts about threats, vulnerabilities, and consequences, and  how these assumptions and other sources of uncertainty affect MSRAM’s  results. For example, to  assess risk in MSRAM, Coast Guard analysts make judgments regarding  such factors as the likelihood of success in interdicting an attack and the  number of casualties expected to result from an attack. Coast Guard  officials agreed with the importance of documenting and communicating  the sources and implications of uncertainty for MSRAM’s risk estimates,  and noted that they planned to develop this documentation as part of an  internal MSRAM verification, validation, and accreditation (VV&A) process  that they expect to complete in the fall of 2011. National Research Council of the National Academies, Review of the Department of  Homeland Security’s Approach to Risk Analysis. Coast Guard Has Used a Risk-Informed Approach to Manage Maritime Security Risk, but Challenges Hinder Sector Efforts

		MSRAM Informs Several National-Level Risk Management Efforts

MSRAM is a security risk analysis and risk management tool and the  Coast Guard intends for it to be used to inform risk management  decisions at all levels of command. MSRAM Has Informed Some Local-Level Risk Management Efforts, but Its Use Has Been Limited by Several Factors

MSRAM has been used to inform a variety of efforts at the sector level,  such as strategic planning, communication with port stakeholders, and  operational and tactical decision making, but its use for operational and  tactical risk management efforts has been limited by a lack of staff time,  the complexity of the MSRAM tool, and competing mission demands,  among other factors. Providing training. MSRAM has the capability of informing operational, tactical, and resource  allocation decisions at all levels of a sector, but the Coast Guard has  generally provided MSRAM training to a limited number of sector staff  with specific MSRAM risk assessment responsibilities, such as port  security specialists, rather than sector staff who may have command or  management responsibilities where MSRAM may apply. Such training on how MSRAM can be used at all levels of  command for risk-informed decision making—including how MSRAM can  assist with the selection of different types of security measures to address  areas of risk and the evaluation of their impacts—could further the Coast  Guard’s efforts to implement its risk management framework and meet its  goal to institutionalize MSRAM as the risk management tool for maritime  security. Coast Guard Measures Risk Reduction but Has Faced Challenges Using This Measure to Inform Decisions

		Coast Guard Developed a Measure to Report Performance in Reducing Risk

The Coast Guard developed a performance measure and supporting  model to measure and report its overall performance in reducing maritime  security risk. This measure identifies the percentage reduction of maritime  security risk, subject to Coast Guard influence, resulting from various  Coast Guard actions. For example, given the inherent uncertainties in estimating risk  reduction, it is unclear if a measure of risk reduction would provide  meaningful performance information for tracking progress against goals  and performance over time. Reliability refers to the precision with which performance is measured,  while validity is the extent to which the measure adequately represents  actual performance. Although the  Coast Guard has taken steps to improve the quality of the supporting  model to provide a more accurate measure, estimating risk reduction is  inherently uncertain and this measure is based on largely subjective  judgments of Coast Guard personnel, and therefore the risk reduction  results reported by the Coast Guard are not based on measurable or  observable activities. Using a risk  reduction measure that more accurately reflects performance  effectiveness can give Coast Guard leaders and Congress a better sense  of progress toward goals, which can support efforts to identify areas for  improvement. Taking these steps would make the Coast Guard’s risk  management approach even stronger. Recommendations for Executive Action

To help the Coast Guard strengthen MSRAM and better align it with NIPP  risk management guidance, as well as facilitate the increased use of  MSRAM across the agency, we recommend that the Commandant of the  Coast Guard take the following three actions:  (1)  Provide more thorough documentation related to key assumptions  and sources of uncertainty within MSRAM and inform users of any  implications for interpreting the results from the model. (2)  Make MSRAM available to appropriate parties for additional external  peer review.
Why GAO Did This Study

Since the terrorist attacks of September 11, 2001, the nation's ports and waterways have been viewed as potential targets of attack. The Department of Homeland Security (DHS) has called for using risk-informed approaches to prioritize its investments, and for developing plans and allocating resources that balance security and the flow of commerce. The U.S. Coast Guard--a DHS component and the lead federal agency responsible for maritime security--has used its Maritime Security Risk Analysis Model (MSRAM) as its primary approach for assessing and managing security risks. GAO was asked to examine (1) the extent to which the Coast Guard's risk assessment approach aligns with DHS risk assessment criteria, (2) the extent to which the Coast Guard has used MSRAM to inform maritime security risk decisions, and (3) how the Coast Guard has measured the impact of its maritime security programs on risk in U.S. ports and waterways. GAO analyzed MSRAM's risk assessment methodology and interviewed Coast Guard officials about risk assessment and MSRAM's use across the agency.

What GAO Found

MSRAM generally aligns with DHS risk assessment criteria, but additional documentation on key aspects of the model could benefit users of the results. MSRAM generally meets DHS criteria for being complete, reproducible, documented, and defensible. Further, the Coast Guard has taken actions to improve the quality of MSRAM data and to make them more complete and reproducible, including providing training and tools for staff entering data into the model. However, the Coast Guard has not documented and communicated the implications that MSRAM's key assumptions and other sources of uncertainty have on MSRAM's risk results. For example, to assess risk in MSRAM, Coast Guard analysts make judgments regarding such factors as the probability of an attack and the economic and environmental consequences of an attack. These multiple judgments are inherently subjective and constitute sources of uncertainty that have implications that should be documented and communicated to decision makers. Without this documentation, decision makers and external MSRAM reviewers may not have a complete understanding of the uses and limitations of MSRAM data. In addition, greater transparency and documentation of uncertainty and assumptions in MSRAM's risk estimates could also facilitate periodic peer reviews of the model--a best practice in risk management. MSRAM is the Coast Guard's primary tool for managing maritime security risk, but resource and training challenges hinder use of the tool by Coast Guard field operational units, known as sectors. At the national level, MSRAM supports Coast Guard strategic planning efforts, which is consistent with the agency's intent for MSRAM. At the sector level, MSRAM has informed a variety of decisions, but its use has been limited by lack of staff time, the tool's complexity, and competing mission demands, among other things. The Coast Guard has taken actions to address these challenges, but providing additional training on how MSRAM can be used at all levels of sector decision making could further the Coast Guard's risk management efforts. MSRAM is capable of informing operational, tactical, and resource allocation decisions, but the Coast Guard has generally provided MSRAM training only to a small number of sector staff who may not have insight into all levels of sector decision making. The Coast Guard developed an outcome measure to report its performance in reducing maritime risk, but has faced challenges using this measure to inform decisions. Outcome measures describe the intended result of carrying out a program or activity. The measure is partly based on Coast Guard subject matter experts' estimates of the percentage reduction of maritime security risk subject to Coast Guard influence resulting from Coast Guard actions. The Coast Guard has improved the measure to make it more valid and reliable and believes it is a useful proxy measure of performance, noting that developing outcome measures is challenging because of limited historical data on maritime terrorist attacks. However, given the uncertainties in estimating risk reduction, it is unclear if the measure would provide meaningful performance information with which to track progress over time. In addition, the Coast Guard reports the risk reduction measure as a specific estimate rather than as a range of plausible estimates, which is inconsistent with risk analysis criteria. Reporting and using outcome measures that more accurately reflect mission effectiveness can give Coast Guard leaders and Congress a better sense of progress toward goals.

What GAO Recommends

GAO recommends that the Coast Guard provide more thorough documentation on MSRAM's assumptions and other sources of uncertainty, make MSRAM available for peer review, implement additional MSRAM training, and report the results of its risk reduction performance measure in a manner consistent with risk analysis criteria. The Coast Guard agreed with these recommendations.