NWS Initiated a 2-Year Prototype and Then Suspended It

Following the Corporate Board’s decision to proceed, NWS initiated the  first of three phases of its prototype, which it planned to continue over the  next 2 years. During the second phase, which NWS expected to take place between  August 2007 and February 2008, the agency planned to have 20 weather  offices share forecasting responsibilities in 2-office pairs. On March  23, 2007, Commerce’s Under Secretary for Oceans and Atmosphere  directed NWS’s director to place all activities associated with NWS’s  concept of operations on hold pending a review by the Deputy Under  Secretary. Moreover, the prototype plan identifies qualitative benefits of the new  concept of operations—including increased efficiency and time for other  activities (such as outreach and research), the ability to switch to a  backup site more quickly when a weather forecast office is disabled due to  loss of power or communications, and an improved ability to focus on  high-impact events. By not conducting a cost-benefit analysis before beginning the prototype,  NWS lacked assurance that its approach was cost-effective. If NWS  decides to proceed with the prototype without conducting a cost-benefit  analysis, the agency will have little basis to ensure that its investment is  sound or that the options it is undertaking provide the greatest return on  investment. NWS Lacks a Rigorous Evaluation Plan for Assessing Its Prototype

In its prototype plan, NWS identified goals and selected measures to  evaluate during its prototype activities, but it did not establish a rigorous  evaluation plan. Specifically, the agency had not defined a comprehensive  set of measures, how it planned to compare prototype results with  baseline performance, or how its measures supported the goals of the  prototype. If NWS were to proceed with its prototype without a rigorous evaluation  plan, the agency would run an increased risk that it could not sufficiently  measure the impact of planned changes on its performance, and would  subsequently make decisions affecting the nation’s weather service on the  basis of incomplete and flawed data. Looking forward, an NWS official stated that they also would ensure that  there would be no degradation of service if the Corporate Board decided  to implement this alternative concept of operations on a national basis  after the prototype was completed. Agency officials explained that the  agency would use its standard procedures for testing and validating  systems and software to ensure that there would be no degradation of  service before moving them into operations. Furthermore, the agency involved  internal stakeholders in planning for its prototype, but it did not seek input  from external stakeholders or establish a plan for stakeholder involvement  throughout the prototype. If  NWS decides to proceed with its prototype, we recommend the Secretary  direct the Assistant Administrator for the National Weather Service to    develop a cost-benefit analysis for the clustered peer approach before  implementing the prototype in any weather forecasting office;   develop an evaluation plan that includes a comprehensive set of measures  that are linked to prototype goals and identifies the baseline performance  that the prototype will be compared with before implementing the  prototype in any weather forecasting office; and   develop a plan for internal and external stakeholder involvement, which  includes a list of relevant stakeholders, roles and responsibilities for these  stakeholders, and a schedule for when stakeholders should be involved  before implementing the prototype in any weather forecasting office. Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) determine the current status of and plans for  the prototype, (2) evaluate whether the justification for the prototype is  sufficient, (3) determine whether the National Weather Service’s (NWS)  plans to evaluate the prototype are adequate, (4) evaluate whether NWS is  sufficiently involving stakeholders in its prototype plans, and   (5) determine how NWS plans to ensure that there will be no degradation  of service during and after the prototype. The NOAA   Global Systems Division office in Boulder was chosen because of its  participation in the first laboratory exercise. NWS established a program  manager to oversee the prototype.
Why GAO Did This Study

Using advanced systems and trained specialists located in 122 weather forecast offices throughout the country, the National Weather Service (NWS) provides storm and flood warnings and weather forecasts to protect life and property and to enhance the national economy. To improve the efficiency of its operations, in November 2006, NWS approved an effort to develop a prototype of an alternative way of operating. Under this prototype, weather forecasting offices would share selected responsibilities. GAO (1) determined the status of and plans for the prototype, (2) evaluated whether the prototype's justification was sufficient, (3) determined whether NWS's plans to evaluate the prototype are adequate, (4) evaluated whether NWS is sufficiently involving stakeholders in its prototype plans, and (5) determined how NWS plans to ensure that there will be no degradation of service during and after the prototype. To do so, GAO analyzed agency documentation and interviewed program officials and stakeholders.

What GAO Found

NWS's prototype is currently on hold pending a reevaluation of the agency's approach. The agency had just begun the first phase of its three-phased prototype to demonstrate a new concept of operations over a 2-year period when, in late March 2007, the Department of Commerce's Under Secretary suspended the prototype because of concerns about the agency's approach. In the first phase, NWS established a program manager and began planning for the next two phases. During the remaining phases, NWS planned to have 20 weather forecasting offices share responsibilities in 2-office pairs and then in 4-office clusters. NWS then planned to decide whether to implement the new concept of operations on a national basis. The justification for the prototype was not sufficient. Before the prototype was suspended, the agency had approved moving forward with its prototype without conducting a cost-benefit analysis. NWS estimated that the prototype would cost approximately $9.3 million and would offer qualitative benefits, such as increased efficiency and an improved ability to focus on severe weather events, but did not quantify benefits or the expected return on its investment. If NWS were to proceed with the prototype without a cost-benefit analysis, it would lack assurance that its approach would be a cost-effective investment for the agency. NWS identified goals and selected measures to evaluate during its prototype activities, but it did not establish a rigorous evaluation plan. Specifically, NWS did not define a full set of needed measures, how it planned to compare prototype results with baseline performance, or how its selected measures supported the prototype goals. If NWS were to proceed without a rigorous evaluation plan, the agency would run an increased risk of not sufficiently measuring the impact of changes on its performance and could make decisions affecting the nation's weather on the basis of incomplete or flawed data. Although NWS involved internal stakeholders in planning its prototype, it did not (1) involve external stakeholders or (2) establish a plan that identified key stakeholders, the stakeholders' responsibilities, and a time line for involving stakeholders and addressing their comments. If NWS were to proceed with its prototype without such a plan, it could not ensure that stakeholder interests would be identified and addressed. NWS planned to mitigate the risk of degradation during and after the prototype by conducting laboratory exercises to understand the impact of the prototype, using a dedicated workstation in each office so that the office could switch to original systems if warranted, and monitoring its systems and products during the prototype. NWS officials stated that should the agency decide to implement the prototype on a national basis, it has standard procedures for testing and validating systems and software to avoid any degradation of service.