Leading Organizations and Experts Identified Key Types and Attributes of Information Security Measures

Leading organizations and experts have identified different types of  measures that are useful in helping to achieve information security goals. Leading Organizations and Experts Identified Key Practices for Developing and Using Information Security Measures

Leading organizations and experts from whom we obtained input  indicated that the successful development of information security  measures depends on adherence to a number of key practices, including  focusing on risks, involving stakeholders, assigning accountability, and  linking to business goals. Presenting these may be appropriate for  information security managers but not for higher-level executives. Agency Information Security Measures and Development Processes Have Not Always Fully Adhered to Key Practices

Federal agencies’ information security performance measures and their  processes for developing them have not always followed key practices  identified by leading organizations. While agencies have developed  measures that fall into each of the three major types (i.e., compliance,  control effectiveness, and program impact), on balance they have relied  primarily on compliance measures, which have a limited ability to gauge  program effectiveness. 3). To the extent that agencies do not use quantifiable  measures of their security control activities, they may limit their ability to  produce accurate and useful assessments of their information security  programs. Agency Measures Were Not Linked to Business Goals  Leading organizations and NIST have stated that security measures need  to be linked to an organization’s overall business priorities to demonstrate  their importance in accomplishing the organization’s mission. Further, OMB’s  annual report to Congress on information security also does not reflect  key practices for communicating the effectiveness of an information  security program. As a result, OMB is limited in its ability to report on the  effectiveness of agency information security programs. In addition, a measure should be collected only if it is useful in  the decision-making process. Until OMB  begins to collect effectiveness measures and report their results through  key practices such as tailoring measures to the audience; correlating data  to derive greater meaning; and capturing progress, trends, and deficiencies  of security controls, the utility of its reports to Congress on the  effectiveness of federal information security programs will be limited. The specific data elements  that OMB required agencies to report have been largely inadequate to  measure the effectiveness of federal information security programs, and  OMB has not sufficiently used key practices, such as correlating the data  and discussing trends and weaknesses, that would have provided a more  complete and valuable assessment. Direct agency CIOs to ensure that all of their measures exhibit the four  key attributes of a measure (i.e., that it be measurable, meaningful,  repeatable and consistent, and actionable). To improve OMB’s process for collecting measures and reporting to  Congress on the status of information security programs, we are  recommending that the Director of OMB take the following two actions:    Revise annual reporting guidance to agencies to require (1) reporting on a  balanced set of measures, including measures that focus on the  effectiveness of control activities and program impact, and (2) inclusion of  all key attributes in the development of measures. Appendix I: Objectives, Scope, and Methodology

The objectives of our review were to (1) describe key types and attributes  of performance measures, (2) identify the practices of leading  organizations for using measures to guide and monitor their information  security control activities, (3) identify what measures federal agencies use  to guide and monitor their information security control activities and how  they are developed, and (4) identify the effectiveness of the measures- reporting practices that the federal government uses to inform Congress  about the effectiveness of information security programs. All were prominent, nationally known  organizations. To determine the effectiveness of the federal government’s practices for  reporting performance measures, we reviewed prior GAO reports and  relevant laws and guidance such as the Federal Information Security  Management Act of 2002 (FISMA) to identify mandatory and optional  practices for reporting information security program information  (including performance measurement information) to the Office of  Management and Budget (OMB) and Congress.
Why GAO Did This Study

Information security is a critical consideration for federal agencies, which depend on information systems to carry out their missions. Increases in reports of security incidents demonstrate the urgency of adequately protecting the federal government's data and information systems. Agencies are required to report to the Office of Management and Budget (OMB) on their information security programs, and OMB is to report results to Congress. Agencies have reported progress in carrying out their activities and have used a variety of measures as the basis of that reporting. GAO was asked to (1) describe key types and attributes of performance measures, (2) identify practices of leading organizations for developing and using measures to guide and monitor information security activities, (3) identify the measures used by federal agencies and how they are developed, and (4) assess the federal government's practices for informing Congress on the effectiveness of information security programs. To do this, GAO met with leading organizations, consulted with experts, and reviewed major federal agencies' policies and practices.

What GAO Found

Experts and leading organizations (nationally known organizations, academic institutions, and state agencies with enterprisewide information security measurement programs) have identified key types and attributes of successful information security measures. These measures fell into three major types: (1) compliance with policies, standards, or legal and regulatory requirements; (2) effectiveness of information security controls; and (3) overall impact of an organization's information security program. Experts and leading organizations also identified four key attributes of successful measures. Specifically, measures should be quantifiable, meaningful (i.e., have targets for tracking progress, be clearly defined, and be linked to organizational priorities), repeatable and consistent, and actionable (i.e., be able to be used to make decisions). Practices of leading organizations for developing measures emphasized the importance of focusing on the risks facing the organization, involving stakeholders from the beginning of the development process, assigning accountability for results, and linking information security programs to overall business goals. Key practices for using the resulting measurements include tailoring information to specific audiences (e.g., senior executives or unit managers); correlating measures to better assess outcomes; and reporting on the progress, trends, and weaknesses revealed by the collected data. Federal agencies have tended to rely on compliance measures for evaluating their information security controls and programs. The measures developed by agencies have not always exhibited the key attributes identified by leading organizations, and agencies have not always followed key practices in developing their measures, such as focusing on risks. To the extent that agencies do not measure the effectiveness and impact of their information security activities, they may be unable to determine whether their information security programs are meeting their goals. OMB's process for collecting and reporting on agency information security programs employs key practices identified by leading organizations and experts but is lacking in some areas. Specifically, many of the measures that OMB requires have key attributes such as being quantifiable, having targets, and being repeatable and consistent, but others do not. Further, OMB's process for collecting information from agencies relies on measures that do not demonstrate the effectiveness of control activities or the impact of information security programs. In addition, OMB does not adequately tailor its reporting for its congressional audience, correlate the data it collects, or discuss trends and weaknesses in information security controls and programs. Until OMB collects measures of the effectiveness of information security programs and appropriately reports the results, Congress will be hindered in its assessment of federal agencies' information security programs.