VA’s mission is to promote the health, welfare, and dignity of all veterans in recognition of their service to the nation by ensuring that they receive medical care, benefits, social support, and lasting memorials. In carrying out this mission, the department manages one of the largest health care delivery systems in the United States that provides enrolled veterans with a full range of services. These services may include primary care; mental health care; and outpatient, inpatient, and residential treatment. VHA, one of the department’s three major components, is responsible for overseeing the provision of health care at all VA medical facilities. IT is widely used and critically important to supporting the department in delivering health care to veterans. As such, VA operates and maintains an IT infrastructure that is intended to provide the backbone necessary to meet the day-to-day operational needs of its medical centers and other critical systems supporting the department’s mission. The infrastructure is to provide for data storage, transmission, and communications requirements necessary to ensure the delivery of reliable, available, and responsive support to all VA staff offices and administration customers, as well as veterans. Over nearly 2 decades, VA pursued multiple efforts to modernize VistA. However, these efforts were abandoned due to expectations of high costs and challenges to ensuring interoperability of health data. Beginning in December 2013, the department initiated VistA Evolution, a joint program between OIT and VHA that focused on implementing a collection of projects to improve the efficiency and quality of veterans’ health care. Specifically, it focused on modernizing the VistA system, increasing the department’s data exchange and interoperability with DOD and private sector health care partners, and reducing the time it takes to deploy new health information management capabilities. The VistA 4 Roadmap was the key plan that the department used to guide VistA Evolution. According to this plan, VistA Evolution was intended to result in lower costs for system upgrades, maintenance, and sustainment. As part of VistA Evolution, the department initiated work to, among other things, standardize VistA instances; expand the use and functionality of the Joint Legacy Viewer; and release enhancements to legacy scheduling, pharmacy, and immunization systems. For example, one focus of the VistA Evolution program over the last several years was to standardize a core set of the system’s modules which, according to the department, account for about 60 percent of VistA. As part of these efforts, the department implemented a process to assess variances in the system at individual sites. According to OIT officials, this process led to more standardization of the code, where possible, and also allowed sites to apply for a waiver if there was a need to continue to operate a nonstandardized VistA instance. Although VistA Evolution was intended to modernize aspects of the system through December 2018, the planned scope of work was reduced as VA redirected the department’s efforts. Specifically, in June 2017, the former VA Secretary announced a significant shift in the department’s approach to modernizing VistA. Rather than continue to use the system, the Secretary stated that the department planned to acquire the same EHR system that DOD is acquiring—Cerner Millennium. According to the department, it has chosen to acquire this product because Cerner Millennium should allow the entire department’s and DOD’s patient data to reside in one system, thus, potentially reducing or eliminating the manual and electronic exchange and reconciliation of data between two separate systems. Accordingly, the department awarded an indefinite delivery, indefinite quantity contract to Cerner in May 2018 for a maximum amount of $10 billion over 10 years. Cerner is to replace the 130 instances of VistA with a standard COTS system to be implemented across VA. This new system is to support a broad range of health care functions including acute care, clinical decision support, dental care, and emergency medicine. When implemented, the new system will be expected to become the authoritative source of clinical data to support improved health, patient safety, and quality of care provided by VA. The EHRM program is responsible for managing the Cerner contract implementation. As of June 2019, the department had issued eight task orders to Cerner to: provide project management and planning support services, conduct site assessments at the initial operating capability sites, host the Cerner system and supporting data, perform data migration and enterprise interface development, develop a functional baseline, deploy the Cerner system at the initial operating capability sites, analyze, design, and develop a technical baseline, and provide additional interface development. For fiscal year 2019, the program was appropriated about $1.1 billion for planning and managing the transition from VistA to Cerner. VA’s Office of the Deputy Secretary approves spending on EHRM activities according to the appropriation. Further, according to the department, funds are tracked as a major IT investment on the Office of Management and Budget’s Federal IT Dashboard. According to VA documentation, the EHRM program is to provide management support and the infrastructure modernization required to install and operate the new system. Further, the department has estimated that an additional $6.1 billion in funding, above the Cerner contract amount, will be needed to fund additional project management support supplied by outside contractors, government labor costs, and infrastructure improvements over the 10-year contract period. Each VA medical facility is expected to continue using VistA until the new system has been deployed. VA plans to deploy the new EHR system at three initial operating capability sites within 18 months of October 1, 2018, with a phased implementation of the remaining sites over the next decade. The three initial deployment sites, located in the Pacific Northwest, are the Mann- Grandstaff, American Lake, and Seattle VA Medical Centers and related clinical facilities that operate the same instances of VistA. These are the first locations where the system is expected to “go live.” The task order to deploy the Cerner system at the three initial sites provides a detailed description of the steps Cerner needs to take in order to reach initial operating capability at the Mann-Grandstaff site in March 2020, and at the Seattle and American Lake sites in April 2020. According to the schedule, the initial operating capability sites are expected to be operational by July 2020. In 2015, we designated VA health care as a high-risk area for the federal government, and we continue to be concerned about the department’s ability to ensure that its resources are being used cost-effectively and efficiently to improve veterans’ timely access to health care. In part, we identified limitations in the capacity of VA’s existing IT systems, including the outdated, inefficient nature of key systems and a lack of system interoperability, as contributors to the department’s challenges related to health care. In our 2019 update to the high-risk series, we stressed that VA should demonstrate commitment to addressing its IT challenges by stabilizing senior leadership, building capacity, and finalizing its action plan for addressing our recommendations and establishing metrics and mechanisms for assessing and reporting progress. We have also issued numerous reports over the last decade that highlighted the challenges facing VA in modernizing VistA and improving EHR interoperability with DOD. For example, Between July 2008 and January 2010, we issued a series of reports related to provisions included in the National Defense Authorization Act for Fiscal Year 2008 that required VA and DOD to, among other things, jointly develop and implement fully interoperable EHR systems or capabilities and establish an Interagency Program Office to be a single point of accountability for their efforts. These reports summarized progress made over time to set up the program office, but also noted that the office was not positioned to function as a single point of accountability for the delivery of the future interoperable capabilities that the departments were planning. In March 2011, the Secretaries of VA and DOD committed the two departments to the development of a new common integrated electronic health record (iEHR) system and, in May 2012, announced their goal of implementing it across the departments by 2017. However, in February 2014, we reported on the departments’ decision to abandon their plans for the iEHR. Specifically, we reported that the Secretaries of VA and DOD, citing challenges in the cost and schedule for developing the iEHR, had announced that they would not continue with the new system and would, instead, pursue separate efforts to modernize or replace their existing systems and work to ensure interoperability between them. Further, we reported that the departments had not addressed management barriers to effectively collaborate on their joint health IT efforts. We made recommendations regarding, among other things, developing a plan to describe the schedule, cost, and roles and responsibilities for the organizations within VA and DOD involved in acquiring, developing, and implementing the EHR systems. The departments agreed with these recommendations and took steps to address them. We reported in August 2015 that VA and DOD, with guidance from the Interagency Program Office, had taken actions to increase interoperability between their EHR systems. However, the office had not yet specified outcome-oriented metrics and established related goals that are important to gauging the impact that interoperability capabilities have on improving health care services for shared patients. As a result, we made several recommendations to VA and DOD to address these deficiencies and the departments agreed with them. VA, DOD, and the Interagency Program Office subsequently took actions that addressed the recommendations. In a June 2018 testimony, we noted that VA had undertaken important analyses to better understand the scope of VistA and identify capabilities that can be provided by the Cerner system. The department also had other key activities underway, such as establishing program governance and EHRM program planning. We noted that critical success factors could serve as a model of best practices that VA could apply to enhance the likelihood that the acquisition of the new system would be successfully achieved. Further, in a September 2018 testimony, we summarized our previously reported findings on the establishment and evolution of the DOD/VA Interagency Program Office, which has been involved in various approaches to increase health information interoperability between the departments. We noted that the office had not been effectively positioned to function as the single point of accountability for the departments’ EHR system interoperability efforts as called for in the National Defense Authorization Act for Fiscal Year 2008. As a result of these findings, we recommended that VA clearly define the role and responsibilities of the Interagency Program Office within the governance plans for acquisition of the department’s new EHR system. The department agreed with the recommendation and stated that the Joint Executive Council, a joint governance body comprised of leadership for both VA and DOD, had approved a role for the office. However, as of June 2019, additional work was ongoing to clarify the role of the Interagency Program Office in VA’s EHR acquisition. In order to maintain internal control activities over an IT system and its related infrastructure, organizations should be able to define physical and performance characteristics of the system, including descriptions of the components and the interfaces. Further, consistent with GAO’s Cost Estimating and Assessment Guide, a comprehensive system definition should identify customization and the environment in which the system operates. While defining a complex IT system can be challenging, having an adequate understanding of its characteristics will better position the organization to comprehensively project and account for costs over the life of a system or program as well as identify specific technical and program risks. Definition of VistA remains important because VA plans to continue using the system during the department’s decade-long transition to the Cerner system. VA maintains multiple documents and a database that describe parts of VistA, including various components and interfaces. However, despite these existing sources, OIT officials acknowledged that there is no comprehensive definition of the VistA system. Consequently, VA has completed a number of efforts to better define VistA and understand the environment in which it operates and additional work is planned in the future. Specifically, VA has documented descriptions of the system, including the components that comprise it. These descriptions are documented in multiple sources: the VA Monograph, VA Systems Inventory, and VA Document Library. The VA Monograph is a document maintained by OIT that provides an overview of VistA and non-VistA applications used by VHA. According to VHA officials, the VA Monograph is the primary document that describes the components of the system. The Monograph describes VistA in terms of modules. For modules identified, including VistA modules, information such as the associated business functions, VA Systems Inventory identification number, and a link to the VA Document Library for additional technical information are provided. The VA Systems Inventory is a database maintained by OIT that identifies current IT systems at the department, including systems and interfaces related to VistA. For systems identified, the database includes information such as the system name, the system status (i.e., active, in development, or inactive), and related system interfaces. The VA Document Library is an online resource for accessing documentation (i.e., user guides and installation manuals) on the department’s nationally released software applications, including VistA. VA has also taken steps to further define the system in its efforts to understand VistA and the environment in which it operates. For example, EHRM program officials recognized the need to further understand the customization of VistA components at the various medical facilities and have conducted analyses to do so. These analyses include: Variance analysis: As part of its VistA Evolution program, which has focused on standardizing a core set of VistA functionality, the department implemented a process to compare the instances of VistA installed at sites to the Enterprise Standard version. The results of this analysis allowed the department to assess the criticality of each variance, which is expected to help with VA’s transition to the Cerner system. Module analysis: EHRM program subject matter experts undertook an analysis that involved reviewing and assessing capabilities provided by VistA modules. This analysis enabled department officials to determine whether the capability provided by a VistA module could be provided by the Cerner system, or whether another COTS solution would be required to support this function going forward. Visual mapping: EHRM program officials also directed an analysis that involved developing a notional visual mapping of VA’s health care applications, components, and supporting systems within the health delivery environment. The results of this analysis provided a description of the current state of one instance of VistA and the VA health environment, which is intended to inform the department of possible opportunities for business process and IT improvements as it proceeds with the Cerner acquisition. Nevertheless, even with these analyses, VA has not yet fully defined VistA, including, for example, identifying performance characteristics of the system and describing the environment in which it operates. The department’s three sources that describe VistA and the additional analyses undertaken do not provide insight into site specific customizations of the system. For example, the VA Monograph does not include information on module customization at local facilities. In addition, according to OIT officials, the systems inventory does not reflect differences among the 130 different instances of VistA and does not take into consideration regional and local customizations of related components. Further, the visual mapping analysis noted that there was not full insight of the intertwined structure of data and applications or the various local customizations of VistA. EHRM program officials stated that they have not been able to fully define VistA and understand all local customizations due to the decentralization of the development of the system and its evolution over more than 30 years. They explained that VistA’s complexity is partly due to the various instances of the system, compounded by local customizations, which have resulted in differences in VistA instances operating at various facilities. According to EHRM program documentation, Cerner’s contract calls for the company to conduct comprehensive assessments to capture the current state of technical and clinical operations at specific facilities, as well as identify site-specific requirements where the Cerner system is planned to be deployed. As of June 2019, Cerner had completed site assessments for the three initial operating capability sites in the Pacific Northwest and had planned additional assessments at future deployment sites. The initial site assessments included, among other things, an assessment of the unique VistA instances and the environment in which the system operates. The continuation of planned site assessments should provide a thorough understanding of the 130 VistA versions, help the department better define VistA, and position it for transitioning from VistA to Cerner’s COTS solution. When using public funds, an agency must employ effective management practices in order to let legislators, management, and the public know the costs of programs and whether they are achieving their goals. To make those evaluations for a program or for a system as large and complex as VistA, a complete understanding of the system and reliable cost information is required. By following a methodology and utilizing reliable data, an agency can ensure that all costs are fully accounted for, which in turn, better informs management decisions, establishes a cost baseline, and enhances understanding of a system’s performance and return on investment. Fundamental characteristics of reliable costs are that they should be accurate (unbiased, not overly conservative or optimistic), well- documented (supportable with source data, clearly detailed calculations, and explanations for choosing a particular calculation method), credible (identifying any uncertainty or biases surrounding data or related assumptions), and comprehensive (costs are neither omitted nor double counted). Identification of VistA’s costs remains important because VA plans to continue using the system during the department’s transition to the Cerner system over the next decade. VA identified costs for VistA and its related activities adding up to approximately $913.7 million, $664.3 million, and $711.1 million in fiscal years 2015, 2016, and 2017, respectively—for a total of about $2.3 billion over the 3 years. However, of the $2.3 billion, the department was only able to demonstrate that approximately $1 billion of these costs were reliable. The department could not sufficiently demonstrate the reliability of the remaining approximately $1.3 billion of VistA costs that it identified. In addition, VA identified other categories of VistA-related costs, but omitted these costs from the total. Of the $2.3 billion total costs for VistA, VA demonstrated that only approximately $1 billion of these costs were reliable. Specifically, OIT officials identified VistA-related costs within seven categories. The officials were able to sufficiently explain why these categories were included in the development and sustainment costs for VistA and how they were documented by the department; the officials also presented detailed source data for our examination. As a result of our review, we determined that the cost data for these seven categories were accurate, well-documented, credible, and comprehensive and, thus, sufficiently reliable. Table 1 provides a summary of the program costs identified for VistA by OIT and VHA for fiscal years 2015 through 2017 that we determined to be reliable. As shown in the table, VA identified costs for the following seven categories for fiscal years 2015 through 2017: VistA Evolution – The VistA Evolution program costs were associated with VistA strategy, system design, product development, and program management. These costs totaled approximately $549.6 million. Interoperability – The Interoperability program focused on sharing electronic health data between VA and non-VA facilities, including private sector providers and DOD. For example, interoperability costs were associated with architecture, strategy, the Interagency Program Office, product development, and program management. These VistA-related costs totaled approximately $140.2 million. Virtual Lifetime Electronic Record (VLER) Health – This program focused on streamlining the transition of electronic medical information between VA and DOD. These VistA-related costs were associated with product development and program management and totaled approximately $81.2 million. Contracts – Contract costs for VistA Evolution included VHA’s obligations associated with workload management, change management, clinical requirements, and clinical interoperability. These VistA-related costs totaled approximately $202.8 million. Intergovernmental personnel acts – Intergovernmental personnel acts are agreements for the temporary assignment of personnel between the federal, state, and local governments; colleges and universities; Indian tribal governments; federally funded research and development centers; and other eligible organizations. These costs accounted for VHA’s need to use outside experts from approved entities for limited periods of time to work on VistA Evolution assignments. The total VistA-related costs were approximately $2.4 million. Memorandums of understanding – According to VHA, memorandums of understanding are agreements used by the administration to obtain the services of personnel between VA entities for VistA-related activities. These agreements accounted for approximately $2.3 million. Pay – Costs in this category included salaries for VHA staff who worked on VistA-related projects as well as travel, training, and supply costs associated with employment. These costs totaled approximately $34.1 million. However, VA was not able to sufficiently demonstrate the reliability of approximately $1.3 billion in costs related to VistA. Specifically, OIT officials identified the additional legacy VistA costs of $1.3 billion that generally fell into three categories: Legacy VistA: Infrastructure, hosting, and system sustainment – Legacy VistA costs are generally related to the maintenance of fully operational items, such as VistA Imaging and Fileman—two key components related to VistA’s operation. The costs also included obligations for costs related to hosting health data in both VA and non-VA facilities. The OIT officials and subject matter experts estimated these total costs to be approximately $343 million during fiscal years 2015 through 2017. However, we were not able to determine the reliability of these costs because, for example, source data were not well documented; changes in the cost information provided to us during our review indicated that the cost data may not be credible; and subject matter experts were unclear about how to separate VistA costs from non- VistA costs. Related software – Related software costs are associated with the software supporting or closely integrated with VistA that were identified by EHRM officials, yet not tracked directly for one of the VistA-related programs. Both OIT and VHA identified software licensing costs as VistA-related obligations. The EHRM program reported these costs to be approximately $389 million in total during fiscal years 2015 through 2017. However, we were not able to determine the reliability of the costs in this category for a variety of reasons, including that source data were not well documented. In addition, VA officials were not clear regarding how the total amounts in each category should be divided between OIT and VHA. Given this confusion, we were not able to determine if the costs were fully accurate or credible. OIT personnel (pay and administrative) – According to EHRM officials, OIT does not track labor costs by program. Instead, the department provided estimations of the amount of salaries paid to OIT government staff working on activities such as VistA Evolution, program management, and overall support of VistA and related applications. OIT personnel costs were estimated by the EHRM program office to be approximately $544 million total during fiscal years 2015 through 2017. However, we were not able to determine the reliability of costs in this category because assumptions made for estimating the personnel and salary costs were not well documented and could not be verified. In addition, VA omitted certain VistA costs from the total costs identified for fiscal years 2015, 2016, and 2017. Specifically, VA omitted the following costs: Additional hosting – OIT officials stated that additional costs related to hosting health data by an outside vendor, as well as hosting backup VistA instances at each of the medical center sites, should also be included in the total costs for VistA; however, VA omitted these costs from the total for fiscal years 2015 through 2017. Specifically, according to the officials, calculating costs for these hosting activities requires subject matter experts to identify equipment, space, utilities, and maintenance costs for resources allocated specifically for VistA. However, the department has not yet developed a methodology to calculate the costs. The officials said they were working on identifying a reliable approach for calculating these costs in the future. Data standardization and testing – OIT officials stated that additional costs related to work on clinical terminology mapping and functional testing were not included in the total costs for VistA for fiscal years 2015 through 2017. This work related to mapping existing clinical data to national standards and making updates to VistA or the Joint Legacy Viewer and included mapping data and building test scripts and reports. OIT officials noted that this work had been critical to the VistA Evolution program, but they did not provide actual cost data in this category. The lack of sufficiently reliable and comprehensive costs indicates that the department is not positioned to accurately report the annual costs to develop and sustain VistA. This is due, in part, to the fact that VA has not followed a well-documented methodology that describes how the department determined the total costs for the system. In lieu of a methodology, OIT officials said that leadership and staff from the program took efforts to identify and track the cost components and contracts associated with the system. However, they noted that costs associated with VistA were not all clearly labeled as VistA in an IT system and it was necessary to estimate other costs. The officials were also unable to verify how VistA-related costs were separated from other department costs in all areas and subject matter experts were not consistently familiar with the estimation methods employed and how VistA was defined for the purposes of calculating costs. Further, VA officials noted that they were still working on the best approach to identifying and calculating omitted costs. Without documenting the methodology for what costs are to be included and how they were identified and calculated, VA’s total does not accurately reflect the development and sustainment costs for VistA. As a result, the department, legislators, and the public do not have the comprehensive, reliable information needed to understand how much it actually cost to develop and maintain the system. Further, VA does not have the reliable information needed to make critical management decisions for sustaining the many versions of VistA over the next 10 years until the Cerner system is fully deployed. VA has initiated a number of actions to prepare for the transition from VistA to the Cerner system. These actions include (1) taking steps to establish a program office reporting to senior agency management, (2) forming a governance structure, (3) conducting assessments at initial system deployment sites, (4) preparing program plans, and (5) setting an initial program baseline. These activities represent important initial steps to prepare for the transition to the new system. The program office is working to hire staff and establish a joint governance structure to coordinate with DOD on the departments’ efforts to implement the Cerner system. Strong agency leadership support is a key factor that can increase the likelihood of a program’s success. For example, senior leadership can define a vision for the program and intervene when there are difficulties. Such leadership can come from the establishment of a program office with staff reporting to senior agency management. VA took steps to establish a program office, under the leadership of the VA Deputy Secretary, to support the contract negotiations between the department and Cerner. Toward this end, in January 2018, the department moved the EHRM program office from OIT to directly report to the VA Deputy Secretary. Then, after the contract with Cerner was awarded in May 2018, a new program office—the Office of Electronic Health Record Modernization—was established in June 2018 to plan and implement the EHRM program. The office is intended to coordinate with OIT and VHA leadership—specifically, VA’s CIO and VHA’s Under Secretary for Health—under the direction of an Executive Director. The Executive Director reports directly to the VA Deputy Secretary. Reporting to the Executive Director is the Deputy Executive Director, whose responsibilities include supporting the program’s execution and management, ensuring the program’s direction is in alignment with VA’s desired outcomes, and identifying strategic challenges related to the program. The Office of Electronic Health Record Modernization also includes three management structures: The Chief Medical Office is responsible for overseeing strategy and planning efforts for change management, user testing and training, and business process re-engineering. It also leads communication efforts for the end users and deployment. The Technology and Integration Office is responsible for providing technical leadership, management, and oversight of IT. As such, the office approves technical requirements and supports interoperability with DOD, as well as performs information security, architecture, data migration and management, configuration management, infrastructure engineering, transition and data engineering, and development. The Program Management Office is responsible for, among other things, providing program control support for the scope, schedule, quality, and risk management for the EHRM program; human resources support for the Office of Electronic Health Record Modernization government staff; financial management for operating plans, budgets, cost estimates and reporting; test and evaluation support; and oversight of contracts providing staffing to the EHRM program. As of May 2019, VA was still working to fully staff the Office of Electronic Health Record Modernization. Figure 1 shows the organization of the Office of Electronic Health Record Modernization. According to program officials and the Office of Electronic Health Record Modernization organization chart, the office is expected to be staffed by 289 government employees. These positions are expected to be filled by April 2020 and represent the staff required for the program to achieve its initial operational capability. According to the program’s January 2019 hiring plan, the office had begun its process to reassign staff and hire additional government employees. VA also awarded a contract for program management support. According to EHRM program officials, the support contractor is to supplement the Office of Electronic Health Record Modernization staff with program and project management support, technical support, community care support, and executive support and internal communications, among other areas. The support contractor provides about 370 personnel to deliver project management support. The contractor reported as of January 2019 that it had achieved the following accomplishments, among others: Developed a Project Readiness Assessment Report including roles, schedules, risk, and measures of success within the Chief Medical Office. Developed a survey to identify key clinical priorities for data migration related to patient safety and clinical quality. Coordinated the site visit schedule and logistics with initial operating capability sites and conducted site surveys at eight outpatient clinics. By establishing a program office reporting to the Deputy Secretary, VA has begun to build a framework to demonstrate senior agency management support of the program. Establishing the program office also focuses oversight and program management of the EHRM program. Implementing collaborative governance brings together key agency executives to discuss investment performance and increases accountability. In addition, it is critical for program officials to be actively engaged with stakeholders to ensure the success of a major acquisition. The department has established a governance structure that includes multiple levels of governance bodies and stakeholders. In addition, VA has prepared charters for the governance boards and identified board membership. According to the charters for the governance bodies, the structure is intended to address technical and functional issues, as well as any joint management issues that arise between VA and DOD as both departments implement the Cerner EHR. As of January 2019, the EHRM program governance structure was comprised of a Steering Committee, Governance Integration Board, Functional Governance Board, Technical Governance Board, and EHR Councils. EHRM program officials have stated that the charters for these boards, which describe their membership and responsibilities, will continue to evolve as the program matures. The Steering Committee, the highest board in the program governance structure, advises the VA Secretary on the progress and performance of the EHRM program toward meeting program goals and outcomes and providing strategic direction on program implementation. This committee is chaired by the Deputy Secretary of VA. Voting members of the committee include, among others, the VA CIO and the Under Secretary for Health. According to the draft charter, the Steering Committee is expected to resolve any items that cannot be resolved at the level of the next lower-level board and is to meet at least quarterly. However, as of January 2019, the Steering Committee had not met. According to program officials, other reviews, such as a monthly program review with the Deputy Secretary, beginning in November 2018, have provided executive-level oversight of the EHRM program and have met the purpose of the Steering Committee. The Governance Integration Board is responsible for integrating and communicating efforts across all lower program governance boards (including the Functional Governance Board and the Technical Governance Board) to meet program goals and milestones. The board has three voting members: the Office of Electronic Health Record Modernization Executive Director, the Assistant Deputy Under Secretary for Health, and the Principal Deputy Assistant Secretary for OIT. According to the charter, this board is expected to act as arbitrator between clinical, technical, and budget priorities and adjudicate items that cannot be resolved at the lower-level boards. In addition, the Governance Integration Board serves as the EHRM program Configuration Control Board. According to the charter, the board is to meet on a monthly basis. According to program officials and meeting minutes, as of January 2019, the Governance Integration Board had met six times. The Functional Governance Board is responsible for providing guidance on the functional and business community needs for the EHR modernization efforts. This board interacts with the Technical Governance Board as a functional and business advisor. The Functional Governance Board is chaired by the program office’s Chief Medical Officer and includes members from a variety of VHA functional areas (e.g., nursing, community care, and patient safety). According to the charter, the board is to meet on a biweekly basis and is to provide guidance to address functional decisions escalated from the EHR Councils. According to program officials and meeting minutes, as of January 2019, the Functional Governance Board had met 10 times. The Technical Governance Board is responsible and accountable for all decisions related to EHRM program technical transformation efforts. The board is expected to provide technical decision recommendations and collaborate with DOD and other external partners. The chair of this board is the Office of Electronic Health Record Modernization’s Chief Technology and Integration Officer. Other voting members include an OIT CIO representative and selected technical directors from within the Office of Electronic Health Record Modernization. The board’s draft charter specifies that it is to meet on a biweekly basis. According to EHRM program officials, as of January 2019, the Technical Governance Board had met 16 times. The EHR Councils are working groups comprised of subject matter experts from both clinical and functional (i.e., business) domains that are to work with Cerner to provide input and recommendations for developing and validating standard workflows. As of October 2018, a total of 12 councils had been established to address clinical processes and six councils had been established to address business processes. A total of 121 VHA field office staff and 100 VHA central office staff were appointed to these councils. In addition, the councils have eight planned national workshops and seven planned local workshops. These workshops are ongoing and are expected to be completed by October 2019. According to program officials, the national workshops are intended to establish a national baseline for workflow configuration decisions. The local workshops are to review the national baseline and make integration decisions to suit local needs. Figure 2 depicts the relationships among VA’s EHRM program governance bodies. In addition to the program’s governance, the Secretaries of VA and DOD issued a joint memorandum in September 2018 asserting the need to establish a joint management structure, which could have responsibilities beyond those currently within the purview of the Interagency Program Office. According to the agency officials, the joint management structure will be expected to leverage lessons learned by DOD from its experience in deploying the Cerner system, such as the timing of infrastructure upgrades. Further, in December 2018, the departments chartered a Joint Electronic Health Record Modernization Work Group to assess the departments’ existing EHR modernization strategies and efforts. According to its charter, the work group is also intended to develop and design recommended approaches, processes, and organizational structures to optimize the use of the departments’ resources in pursuit of EHR interoperability objectives. The joint working group is to develop short- and long-term recommendations to support four objectives to provide: a single accountable authority to facilitate decision-making and an organizational structure to support the delivery of a single, coordinated clinical and business workflows; and a coordinated implementation plan and detailed timelines. According to EHRM program officials, the joint working group is to define the joint management structure to be used to coordinate between the departments. According to the charter, the goal is for the recommended joint organization to be operational by the end of September 2019. As previously discussed, according to EHRM program officials, the department determined that site-specific assessments are required to allow Cerner to appropriately identify the requirements for system implementation at each site. To refine the scope of work required for initial operating capability, Cerner and the department conducted assessments, beginning in July 2018, at the three sites identified to be part of the initial operating capability of the program. These site assessments included, among other things, an assessment of the IT infrastructure at each site and identification of site-specific requirements. Additional site assessments are planned at every facility before the Cerner system will be deployed at each location. According to the task order, the assessments are expected to provide perspective on the current state of technical and clinical operations of each facility beyond VA’s current documentation. For example, Cerner is expected to document all interfaces with medical devices, third-party systems and other data sets at each site, as well as update monthly a site readiness checklist to inform comprehensive deployment planning. According to the assessments of the three initial operating capability sites, a number of issues have been identified such as updating or replacing infrastructure and workstations to be compatible with the Cerner COTS system. In addition, according to the site assessments, the services offered by the department, such as telehealth and behavioral health, are generally more expansive than commercial deployments and will require increased collaboration between VA and Cerner to meet business and system requirements. Thus, the assessments are intended to position Cerner and the department to have more information readily available in order to better plan for site-specific issues prior to actual implementation. Program planning is critical for ensuring effective management of key aspects of an IT program and serves as the basis for controlling and managing project performance. These key aspects include, for example, identification of the program’s scope, responsible organizations, costs, and schedules. The Office of Electronic Health Record Modernization Executive Director approved an initial Program Management Plan for the EHRM program in November 2018. According to the plan, it is to be used to guide the management of the EHRM program and defines the program’s policies and processes necessary to achieve the program’s goals. It briefly defines the program’s scope and strategy, including the assumptions made. For example, according to the plan, the EHRM program assumes that VA and DOD will use a single instance of the Cerner system. Further, it states that both the legacy VistA data and EHRM data will be available to both VistA and new system users during the transition. The Program Management Plan also identifies a series of subordinate plans that have been developed to further elaborate on specific program planning and execution activities. For example, the plan summarizes the Deployment Management Plan, which details the strategy and tasks required from initial site assessment through configuration, testing, training, change management, deployment, and transition to sustainment. The plan also describes the Schedule Management Plan, which defines the development and maintenance of the integrated master schedule for the life of the program. Thus, the Program Management Plan provides the guidance for where to look for key planning information for the department. The EHRM program also developed a draft Risk Management Plan, dated September 2018, that defines how risk and issue planning, analysis, and management are to be implemented. The draft risk management process consists of risk identification and mitigation, including conducting risk management planning, identification, analysis, response planning, response identification, and monitoring. According to the plan, management of overall program risk is intended to keep risk exposure within an acceptable range and maximize the likelihood of achieving overall objectives. In addition, the EHRM program developed plans for change management, communications, and training activities to ensure that VA clinicians, staff members, volunteers, and veterans understand and are ready for the changing systems and processes that will impact them. The initial versions of the plans were delivered by Cerner in November 2018. The program’s approach is to continue to evolve these plans as the program matures. By developing these program plans, VA is taking steps to ensure effective management of key aspects of the EHRM program. Baselined program plans act as a guide throughout the life of an investment to provide a basis for measuring performance, identify who is accountable for the deliverables, describe the implementation approach and interdependencies, identify key decisions, and embed quality assurance and reviews. Ultimately, baseline management demonstrates that a project is under financial and managerial control. According to EHRM program officials, on October 30, 2018, the program conducted a review of the time period from contract award through initial operating capability. The review validated the scope of the program for the transition of VistA to the initial operating capability sites, identified an initial work breakdown structure, and included an integrated master schedule and a cost baseline. The results of this review established a baseline for the initial operating capability and changes to the baseline are subject to change control. Also, as a result of the review, the Office of Electronic Health Record Modernization is to conduct monthly program reviews to inform the Deputy Secretary of the status of the EHRM program. According to EHRM program officials, upgrades to the IT infrastructure are to be accomplished by OIT, and the local area network infrastructure is to be upgraded at all initial operating capability sites prior to implementation of the new system. As baselined, upgrades of end user devices are scheduled to be completed at the Mann-Grandstaff site by September 2019, the American Lake site by October 2019, and the Seattle site by November 2019. Program officials have stated that the goal is to have infrastructure upgrades at a site completed 6 months before the site begins to implement the Cerner system. However, in May 2019, EHRM program officials indicated that infrastructure updates may be delayed for the initial sites by up to 3 months. After an evaluation of the initial operating capability, the EHRM program is to determine whether the minimum operational capabilities have been achieved. Figure 3 shows a timeline of the baselined implementation milestones for the initial sites, established at the review held in October 2018. The baseline review also included identifying and addressing program risks related to the Cerner system implementation. The review identified 10 program risks, prioritized the risks by probability and impact, and assigned mitigation plans for the risks. For example, the review identified the risk that if required infrastructure upgrades were not implemented, then VA would not be able to deploy a fully operational EHR system. The program identified development of acquisition strategies to address infrastructure requirements from the site assessments as an action to mitigate this risk. By establishing a program baseline for the initial operating capability, VA has instituted a basis for measuring actual versus planned program performance. In addition, the risk mitigation plans provide an approach to address the identified risks. VA lacks a comprehensive definition of the VistA system that captures the complexity of the system, the environment in which it operates, and the local customizations that have evolved in the VistA instances over many years. Consequently, VA has engaged in efforts to provide additional insight into the system. The department plans to continue to conduct comprehensive site-specific assessments with Cerner to refine its understanding of the unique VistA instances and the environment in which the system operates. The continuation of planned site assessments should help VA better define VistA. With regard to calculating costs for VistA, the department has identified reliable costs for approximately $1 billion in development and sustainment for the system over 3 fiscal years. However, VA was not able to sufficiently demonstrate the reliability of an additional $1.3 billion of costs identified and omitted other relevant costs from the total. The cost deficiencies existed largely because VA officials were uncertain about what to identify as part of VistA; documentation related to certain categories of costs was incomplete; and a documented methodology for identifying and reporting those costs does not exist. As a result, VA lacks the comprehensive and reliable cost information needed to make critical management decisions for sustaining the system and ensuring an accurate basis for reporting on the return on its investment for replacing VistA. VA has taken a number of actions to prepare for the transition from VistA to the Cerner system, such as establishing and beginning to staff a program office, forming a governance structure, conducting site assessments at initial sites, preparing program plans to guide the initial implementation, and setting an initial program baseline to help guide implementation of the system at three key sites. The Secretary of VA should direct the Under Secretary for Health and the Assistant Secretary for Information and Technology/Chief Information Officer to develop and implement a methodology for reliably identifying and reporting the total costs of VistA. The methodology should include steps to identify the definition of VistA and what is to be included in its sustainment activities, as well as ensure that comprehensive costs are corroborated by reliable data. (Recommendation 1) VA provided written comments on a draft of this report. In its comments (reprinted in appendix II), the department generally agreed with our conclusions and concurred with our recommendation. The department stated that it will provide the actions it plans to take to address the recommendation within 180 days. VA also provided technical comments, which we incorporated as appropriate. We are sending copies of this report to the appropriate congressional committees, the Secretary of VA, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staffs have any questions on matters discussed in this report, please contact me at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III. Our objectives were to: (1) determine the extent to which the Department of Veterans Affairs (VA) has defined the Veterans Health Information Systems and Technology Architecture (VistA), (2) evaluate VA’s annual costs to develop and sustain VistA, and (3) describe the actions VA has taken to transition from VistA to the Cerner system. To address the first objective, we examined VA documentation including the VA Monograph, reports from the VA Systems Inventory, and documents listed in the VA Software Document Library. These documents were cited by VA officials as sources that define the VistA system and provide information on modules and interfaces. Our review and compilation of information from these three sources enabled us to describe the various sources used at the department to document information about the VistA system and identify the limitations of each source. We also examined the VistA Product Roadmap, which described modernization plans and achievements related to VistA. Further, we interviewed officials from the Veterans Health Administration (VHA) to obtain information on additional efforts undertaken by the department to further understand and define VistA. In addition, we reviewed program documentation related to three analyses undertaken by VA to further define VistA. These analyses included the department’s efforts to ascertain variances between versions of VistA, identify components of VistA to be replaced by the Cerner System, and document the current state of a sample instance of VistA. For example, we examined VA documentation that described software modules available in the department’s VistA product and program documentation identifying components of VistA to be replaced by the Cerner system. In addition, our review of a visual mapping developed for Electronic Health Record Modernization (EHRM) program officials depicting the environment in which VistA operates allowed us to describe the size and complexity of the system and how it is used by the department. Further, we compared the extent to which VA has defined VistA with criteria for defining information technology (IT) systems described in GAO’s Standards for Internal Control in the Federal Government and our Cost Estimating and Assessment Guide. In addition, we reviewed EHRM program documentation related to site assessments that have taken place at initial operating capability sites and are planned for future sites. Specifically, we reviewed the relevant contract task order to understand how site assessments were planned and to identify site-specific gaps between the current VistA system in use and the target future Cerner system. We supplemented our documentation reviews with information obtained through interviews with officials from VA’s Office of Information and Technology (OIT), VHA, and the EHRM program office. To address the second objective, we examined department documentation of costs associated with the development and sustainment (operation and maintenance) of VistA for fiscal years 2015, 2016, and 2017. These 3 fiscal years were selected because development and sustainment cost information for full fiscal years should have been available during the time period in which we conducted our evaluation. To compile the total costs, we examined all categories of costs identified by VA to determine reliability of the source data. We also discussed the methodology VA used related to identifying costs and estimating costs when source data was not available with officials from the EHRM program. We compared the identified cost data to best practices described in GAO’s Cost Estimating and Assessment Guide that are the basis for effectively capturing reliable program costs. The guide also describes the importance of documenting the methodology by which costs are included and how they are calculated in detail, step by step, to provide enough information so that someone unfamiliar with the program could easily recreate or update cost calculations. Specifically, we analyzed all cost documentation provided by the department over the course of our work. For example, OIT officials identified VistA costs tracked under three programs—VistA Evolution, Interoperability, and Virtual Lifetime Electronic Record (VLER) Health– and VHA officials reported that costs for the system were tracked separately from OIT through various types of contracts and agreements associated with VistA Evolution. In regard to the OIT and VHA program data, VA provided detailed source data that we analyzed for reliability and verified the calculations of costs identified over the course of our work. We also examined the documentation and controls related to the IT systems VA identified as the source of these cost data. The systems included OIT’s Budget Tracking Tool and VA’s Financial Management System. Further, we discussed the nature of the cost data, the rationale behind why each cost line item was included, and any anomalies found during our analysis with cognizant OIT and VHA officials. For example, anomalies included omitted contract numbers or transposed entries in summary tables. As a result of these efforts, OIT and VHA were able to sufficiently demonstrate the reliability of the program data for the purpose of calculating costs for VistA. Officials from the EHRM program also identified costs that were not directly tracked under the program areas previously mentioned. OIT and VHA relied upon subject matter experts or vendors to identify costs or to calculate estimates for cost categories such as sustainment, maintenance, co-location, hosting, pay, administrative, and infrastructure costs related to VistA operations. We analyzed the data provided for reliability consistent with GAO Cost Estimating and Assessment Guide over the course of our work. Further, we discussed the nature of the cost data, the rationale behind why each cost line item was included, and any anomalies found during our analysis with cognizant OIT and VHA officials. We also interviewed OIT and VHA subject matter experts and vendors identified by VA to examine the rationale or methodology for how the costs were identified and estimated. During the course of our work, VA continued to revise these estimates as part of the department’s efforts to identify the costs for VistA and could not provide a consistent, documented methodology for how the costs were calculated or provided only summary costs that could not be analyzed. As such, VA was not able to sufficiently demonstrate the reliability of legacy VistA, related software, and OIT personnel costs for our purpose of calculating the total costs for VistA. This report does not conclude that the data are unreliable, only that a reliability determination could not be made during the course of our work. However, given the importance of these related costs to VistA, we have summarized and reported these costs in the total cost amount for VistA to more accurately approximate the magnitude of total costs, but have not reported itemized costs in these areas. Finally, the department identified that there were additional costs that should be included in the compilation of the total costs for VistA related to additional hosting costs and data standardization and testing. However, the department did not provide such data to include in the total costs for VistA. To address the third objective, we examined the department’s decision memorandums and charters establishing the Office of Electronic Health Record Modernization and the EHRM program to manage VA’s transition from VistA to Cerner. We also examined the statement of work for the program support contract as well as VA’s draft charters, program briefings, and organization charts that describe plans to govern the program to acquire the Cerner system. Specifically, we examined VA’s plans to establish a structure for governing technical and functional issues and joint decisions that arise with the Department of Defense. To understand how site assessments were used to refine the scope of work, we examined the site assessment task order and the site assessment reports. To understand how the program office plans to manage the program, we examined the EHRM Program Management Plan and subordinate plans that guide the management of the program and describe ongoing efforts to define the policies and processes necessary to achieve the program’s goals. To address the program’s establishment of an initial program baseline, we examined the decision memorandum approving the award of the Cerner contract, the briefings presented to program stakeholders at the initial program baseline review, and the documents supporting the program baseline review. We supplemented our analysis with information obtained through interviews with relevant department officials including the Executive Director and Chief Technology and Integration Officer for the EHRM program. We conducted this performance audit from August 2017 to July 2019 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. In addition to the contact named above, Mark Bird (Assistant Director), Jennifer Stavros-Turner (Analyst in Charge), John Bailey, David Blanding, Chris Businsky, Juaná Collymore, Rebecca Eyler, Jacqueline Mai, Scott Pettis, and Charles Youman made key contributions to this report.
VA provides health care services to approximately 9 million veterans and their families and relies on its health information system—VistA—to do so. However, the system is more than 30 years old, is costly to maintain, and does not fully support exchanging health data with DOD and private health care providers. Over nearly 2 decades, VA has pursued multiple efforts to modernize the system. In June 2017, the department announced plans to acquire the same system—the Cerner system—that DOD is implementing. VA plans to continue using VistA during the decade-long transition to the Cerner system. GAO was asked to review key aspects of VistA and VA's plans for the new acquisition of the Cerner system. The objectives of the review were to (1) determine the extent to which VA has defined VistA, (2) evaluate VA's annual costs to develop and sustain VistA, and (3) describe the actions VA has taken to transition from VistA to the Cerner system. GAO analyzed documentation that defines aspects of VistA and identifies components to be replaced; evaluated the reliability of cost data, including obligations associated with the development and sustainment of VistA for fiscal years 2015, 2016, and 2017; and reviewed program documentation related to VA's program, governance, and plans to transition to Cerner. The Department of Veterans Affairs (VA) has various documents and a database that describe parts of the Veterans Health Information Systems and Technology Architecture (VistA); however, the department does not have a comprehensive definition for the system. For example, VA has identified components that comprise VistA, identified interfaces related to the system, and collected system user guides and installation manuals. VA has also conducted analyses to better understand customization of VistA components at various medical facilities. Nevertheless, the existing information and analyses do not provide a thorough understanding of the local customizations reflected in about 130 versions of VistA that support health care delivery at more than 1,500 sites. Program officials stated that they have not been able to fully define VistA due to the decentralization of the development of the system for more than 30 years. Cerner's contract to provide a new electronic health record system to VA calls for the company to conduct comprehensive assessments to identify site-specific requirements where its system is planned to be deployed. Three site assessments have been completed and additional assessments are planned. If these assessments provide a thorough understanding of the 130 VistA versions, the department should be able to define VistA and be better positioned to transition to the new system. VA identified costs for VistA and its related activities adding up to approximately $913.7 million, $664.3 million, and $711.1 million in fiscal years 2015, 2016, and 2017, respectively—for a total of about $2.3 billion over the 3 years. However, of the $2.3 billion, the department was only able to demonstrate that approximately $1 billion of these costs were sufficiently reliable. In addition, the department omitted VistA-related costs from the total. The lack of a sufficiently reliable and comprehensive total cost for VistA is due in part to not following a well-documented methodology that describes how the department determined the costs for the system. As a result of incomplete cost data and data that could not be determined to be sufficiently reliable, the department, legislators, and the public do not have a complete understanding of how much it has cost to develop and maintain VistA. Further, VA lacks the information needed to make decisions on sustaining the many versions of the system. VA has initiated a number of actions to prepare for the transition from VistA to the Cerner system. These actions include taking steps to establish and begin to staff a program office, forming a governance structure, conducting assessments at the initial sites, preparing program plans to guide the initial system implementation, and setting a program baseline to help guide implementation at the initial sites. The department's actions in these important areas are ongoing. Additional actions are in progress to address GAO's September 2018 recommendation that VA clearly define the role and responsibilities of the joint Department of Defense (DOD) and VA Interagency Program Office in the department's governance plans for the new electronic health record system. VA intends to continue maturing and fully establishing a program management organization and a program governance structure to track program progress. GAO is recommending that VA develop and implement a methodology for reliably identifying and reporting the total costs of VistA. VA agreed with the recommendation.