Agencies Use a Variety of Processes to Determine Investments’ CIO Ratings

Agencies determine investments’ CIO ratings using a variety of  processes, which include OMB’s suggested factors. However, their  interpretation of these factors varies significantly. As noted above, we reviewed data reported to OMB as part of the federal  budget process to identify major investments which planned to spend at  least 80 percent of their fiscal year 2015 funding on development,  modernization, and enhancement activities. Each of the 17 agencies has  incorporated at least 2 of OMB’s suggested factors into their CIO’s risk  rating processes and 9 use all of the factors. In  particular, most agencies considered active risks when rating  investments, but others only evaluated compliance with risk processes. Most Selected Agencies Require Monthly CIO Rating Updates

Thirteen agencies’ process guidance calls for at least monthly updates to  CIO ratings, one agency (DHS) schedules its reviews based on risk, and  three agencies require less frequent updates. These provisions require  that agencies report at least semi-annually to OMB on each major IT  investment and include data on cost, schedule, and performance. This resulted in  95 investments at 15 agencies. Our assessments of these investments  generally showed more risk than the associated CIO ratings. In particular,  of the 95 investments we reviewed, our assessments matched the CIO  ratings 22 times, showed more risk 60 times, and showed less risk 13  times. We identified three factors which contributed to these differences:  (1) 40 of the 95 CIO ratings were not updated in April 2015, (2) three  agencies’ rating processes span longer than 1 month, and (3) seven  agencies’ rating processes did not focus on active risks (as previously  discussed). Such updates will help to ensure that the information on the Dashboard is  timely and accurately reflects recent changes. Longer processes mean that CIO ratings are based  upon older data and may not reflect the current level of investment risk. In  all cases, CIO ratings that do not incorporate active risks increase the  chance that ratings do not reflect the true likelihood of investment  success. Recommendations for Executive Action

To better ensure that the Dashboard ratings more accurately reflect risk,  we recommend that:  the Secretaries of the Departments of Agriculture, Education, Energy,  Health and Human Services, the Interior, State, and Veterans Affairs;  and the Director of the Office of Personnel Management direct their  CIOs to factor active risks into their IT Dashboard CIO ratings;  the Secretaries of the Departments of Defense, Education, and  Homeland Security; and the Commissioner of the Social Security  Administration direct their CIOs to update their CIO ratings at least as  frequently as required in OMB’s guidance; and  the Secretaries of the Departments of Agriculture, Commerce,  Defense, Education, Energy, Health and Human Services, Homeland  Security, State, Transportation, the Treasury, Veterans Affairs; the  Administrator of the Environmental Protection Agency; and the  Commissioner of the Social Security Administration direct their CIOs  to ensure that their CIO ratings reflect the level of risk facing an  investment relative to that investment’s ability to accomplish its goals. In written comments, SSA agreed with our recommendation to update  its CIO ratings on a monthly basis. Comments from the agencies to which we did not make  recommendations are discussed in more detail here. Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) describe agencies’ processes for determining  the Chief Information Officer (CIO) risk ratings for major information  technology (IT) investments and (2) assess the risk of federal IT  investments and analyze any differences with the investments’ CIO risk  ratings. According to OMB’s guidance, CIO ratings “should reflect the CIO’s  assessment of the risk and the investment’s ability to accomplish its  goals.” To create our assessments of risk, we combined each  investment’s detailed risk lists, known as risk registers, with several  additional metrics.
Why GAO Did This Study

Although the government spends more than $80 billion in information technology (IT) annually, many of the investments have failed or have been troubled. In December 2014, provisions commonly referred to as the Federal Information Technology Acquisition Reform Act (FITARA) were enacted. Among other things, FITARA states that OMB shall make available to the public a list of each major IT investment including data on cost, schedule, and performance. OMB does so via the Federal IT Dashboard—its public website that reports on major IT investments, including ratings from CIOs which should reflect the level of risk facing an investment.
GAO's objectives were to (1) describe agencies' processes for determining CIO risk ratings for major federal IT investments primarily in development and (2) assess the risk of federal IT investments and analyze any differences with the investments' CIO risk ratings. To do so, GAO selected major IT investments with at least 80 percent of their fiscal year 2015 budget allocated to development (resulting in 95 investments across 15 agencies) and compared CIO rating processes to OMB guidance. GAO also analyzed data on those investments to create its own risk assessments.

What GAO Found

Agencies determined investments' Chief Information Officer (CIO) ratings using a variety of processes, which included the Office of Management and Budget's (OMB) six suggested factors (including risk management, requirements management, and historical performance). Specifically, all 17 selected agencies incorporated at least two of OMB's factors into their risk rating processes and 9 used all of the factors. However, agencies' interpretations of these factors varied. For example, most agencies considered active risks, such as funding cuts or staffing changes, when rating investments, but others only evaluated compliance with the agency's risk management processes. Further, 13 agencies required monthly updates to CIO ratings as does OMB (as of June 2015), 1 agency scheduled its reviews based on risk, and 3 agencies required updates less often than on a monthly basis.
GAO's assessments generally showed more risk than the associated CIO ratings. In particular, of the 95 investments assessed, GAO's assessments matched the CIO ratings 22 times, showed more risk 60 times, and showed less risk 13 times (see graphic).
Forty of the 95 CIO ratings were not updated during the month GAO reviewed, which led to more differences between GAO's assessments and the CIOs' ratings. This underscores the importance of frequent rating updates, which help to ensure that the information on the Dashboard is timely and accurately reflects recent changes to investment status.
Three agencies' rating processes span longer than 1 month. Longer processes mean that CIO ratings are based upon older data and may not reflect the current level of investment risk.
Seven agencies' rating processes did not focus on active risks. According to OMB's guidance, CIO ratings should reflect the CIO's assessment of the risk and the investment's ability to accomplish its goals. CIO ratings that do not incorporate active risks increase the chance that ratings overstate the likelihood of investment success.

What GAO Recommends

GAO is making 25 recommendations to 15 agencies to improve the quality and frequency of CIO ratings. Twelve agencies generally agreed with or did not comment on the recommendations and three agencies disagreed, stating their CIO ratings were adequate. GAO continues to believe these recommendations are valid.