GAO Has Previously Reported on the Dashboard’s Value, Data Quality, and Recent Improvements

We have previously reported that OMB has taken significant steps to  enhance the oversight, transparency, and accountability of federal IT  investments by creating its IT Dashboard, and by improving the accuracy  of investment ratings. CIOs Rated Most IT Investments as Low Risk or Moderately Low Risk; Agencies Had Mixed Results in Reducing Higher Risk Levels

As of March 2012, CIO ratings for most investments listed on the  Dashboard for the six agencies we reviewed indicated either low risk or  moderately low risk (223 out of 313 investments across all the selected  agencies). The figure also reports  agencies’ budgets for their major IT investments for fiscal year 2012, as  presented on the Dashboard. These investments were rated at the same risk level during  every rating period (see fig. In contrast, the other two  agencies—OPM and HHS—changed the CIO rating for more than 70  percent of their investments at least once between the investment’s initial  rating and the rating reported as of March 2012. Two agencies—DHS and OPM—reported more investments with reduced  risk in March 2012, as compared with initial ratings. The other four  agencies reported more investments with increased risk. Include input from stakeholders. Among the agencies we reviewed, DOD was unique in that its ratings  reflected additional considerations beyond OMB’s instructions. CIO Ratings Present Benefits and Challenges for Agencies’ Investment Management

Selected agencies identified various benefits associated with performing  CIO ratings and Dashboard reporting in general. Increased quality of investment performance data. Increased focus on project management practices. Some of these benefits were interrelated. Nevertheless, agencies also identified challenges associated with  producing and reporting CIO ratings. First, three agencies reported a  challenge associated with the time and effort required to gather, validate,  and gain internal approval for CIO ratings and other data reported to the  Dashboard. Its CIO  ratings, in particular, have improved visibility into changes in the risk  levels of agencies’ investments over time. Recommendations for Executive Action

To ensure that OMB’s preparation of the President’s budget submission  accurately reflects the risks associated with all major IT investments, we  are recommending that the Federal CIO analyze agency trends reflected  in Dashboard CIO ratings, and present the results of this analysis with the  President’s annual budget submission. To ensure that DOD’s CIO evaluations of investment risk for its major IT  Dashboard investments reflect all available performance assessments  and are consistent with the department’s own guidance for managing risk,  we are recommending that the Secretary of Defense direct the  department’s CIO to reassess the department’s considerations for  assigning CIO risk levels for Dashboard investments, including  assessments of investment performance and risk from outside the  programs, and apply the appropriate elements of the department’s risk  management guidance to OMB’s evaluation factors in determining CIO  ratings. Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) characterize the Chief Information Officer (CIO)  ratings for selected federal agencies’ information technology (IT)  investments as reported over time on the federal IT Dashboard; (2)  determine how agencies’ approaches for assigning and updating CIO  ratings vary; and (3) describe the benefits and challenges associated with  agencies’ approaches to the CIO rating. Collectively,  these agencies accounted for approximately $51 billion, or 65 percent, of  2011 spending on IT investments. The six agencies are the Department  of Defense, Department of Homeland Security, Department of Health and  Human Services, Department of the Interior, National Science  Foundation, and Office of Personnel Management. To address our second objective, we reviewed available documentation,  obtained written responses to questions we posed to all agencies, and  interviewed OMB and agency officials to determine their policies and  practices related to assigning and updating the CIO ratings and related  data for the Dashboard. Appendix II: Risk Levels for Investments at Selected Agencies, as of March 2012

The table below lists the total number of major information technology (IT)  investments rated on the federal IT Dashboard as of March 2012 for each  agency selected for this review, with the numbers of investments rated at  each of the risk levels specified by the Office of Management and Budget  (OMB) for the chief information officer (CIO) rating.
Why GAO Did This Study

In June 2009, OMB launched the federal IT Dashboard, a public website that reports performance data for over 700 major IT investments that represent about $40 billion of the estimated $80 billion budgeted for IT in fiscal year 2012. The Dashboard is to provide transparency for these investments to aid public monitoring of government operations. It does so by reporting, among other things, how agency CIOs rate investment risk. GAO was asked to (1) characterize the CIO ratings for selected federal agencies' IT investments as reported over time on the Dashboard, (2) determine how agencies' approaches for assigning and updating CIO ratings vary, and (3) describe the benefits and challenges associated with agencies' approaches to the CIO rating.
To do so, GAO selected six agencies spanning a range of 2011 IT spending levels and analyzed data reported for each of their investments on the Dashboard. GAO also interviewed agency officials and analyzed related documentation and written responses to questions about ratings and evaluation approaches, as well as agency views on the benefits and challenges related to the CIO rating.

What GAO Found

Chief Information Officers (CIO) at six federal agencies rated the majority of their information technology (IT) investments as low risk, and many ratings remained constant over time. Specifically, CIOs at the selected agencies rated a majority of investments listed on the federal IT Dashboard as low risk or moderately low risk from June 2009 through March 2012; at five of these agencies, these risk levels accounted for at least 66 percent of investments. These agencies also rated no more than 12 percent of their investments as high or moderately high risk, and two agencies (Department of Defense (DOD) and the National Science Foundation (NSF)) rated no investments at these risk levels. Over time, about 47 percent of the agencies' Dashboard investments received the same rating in every rating period. For ratings that changed, the Department of Homeland Security (DHS) and Office of Personnel Management (OPM) reported more investments with reduced risk when initial ratings were compared with those in March 2012; the other four agencies reported more investments with increased risk. In the past, the Office of Management and Budget (OMB) reported trends for risky IT investments needing management attention as part of its annual budget submission, but discontinued this reporting in fiscal year 2010.
Agencies generally followed OMB's instructions for assigning CIO ratings, which included considering stakeholder input, updating ratings when new data become available, and applying OMB's six evaluation factors. DOD's ratings were unique in reflecting additional considerations, such as the likelihood of OMB review, and consequently DOD did not rate any of its investments as high risk. However, in selected cases, these ratings did not appropriately reflect significant cost, schedule, and performance issues reported by GAO and others. Moreover, DOD did not apply its own risk management guidance to the ratings, which reduces their value for investment management and oversight.
Various benefits were associated with producing and reporting CIO ratings. Most agencies reported (1) increased quality of their performance data, (2) greater transparency and visibility of investments, and (3) increased focus on project management practices. Agencies also noted challenges, such as (1) the effort required to gather, validate, and gain internal approval for CIO ratings; and (2) obtaining information from OMB to execute required changes to the Dashboard. OMB has taken steps to improve its communications with agencies.

What GAO Recommends

GAO is recommending that OMB analyze agencies' investment risk over time as reflected in the Dashboard's CIO ratings and present its analysis with the President's annual budget submission, and that DOD ensure that its CIO ratings reflect available investment performance assessments and its risk management guidance. Both OMB and DOD concurred with our recommendations.