Since 2002, OPM has used a telework survey—the data call—to annually  collect information from the executive agencies in order to provide  Congress with a report on the status of telework across these agencies. OPM Revised the 2011 Telework Data Call to Prepare for Reporting under the Telework Enhancement Act of 2010

In response to the Telework Enhancement Act of 2010, OPM revised the  2011 data call and provided instructions to executive agency respondents  that incorporated common definitions and standards to use in providing  OPM with their agency data. While  some of the information provided at the two training sessions was similar,  each session contained some new information usually in response to  questions raised at a previous session. If  OPM reports agency telework progress based on data collected using  definitions revised year to year, OPM may reach erroneous conclusions,  although OPM officials have said they are taking steps to try and prevent  this possibility. OPM  officials said that they will need to indicate this in their report to Congress. While OPM made changes to the 2011 data call to allow it to meet some  of its reporting requirements under the act and better assist agencies in  responding to the data call, our analysis found the 2011 data call did not  fully meet some generally accepted survey standards. Since 2003, OPM has consistently expressed concerns about  the methods agencies use to collect telework data. Executive agencies provide telework  participation and frequency data using a variety of methods, such as  relying on estimates, counting telework agreements, and using automated  time and attendance records to track telework participation. According OPM, agencies will begin piloting these automated data  collection systems for the 2012 telework data call. However, continuous improvement efforts sometimes result  in a trade-off between the desire for data consistency and a need to  improve data collection and maintenance of a consistent data series over  time. With the planned change to the  method of data collection, it may not be possible to compare the 2011  data to future data. However, these officials said that the 2011 data,  notwithstanding its limitations, will be useful in identifying and  understanding any major agency changes in reported participation that  could occur during a transition to automated data collection. In addition, variation in  training sessions and OPM’s uncertainty as to whether all respondents  attended training, could lead to respondents’ potential misunderstanding  of important terms and instructions. The validity and reliability of the  reported 2011 telework data for some of the responding agencies may be  questionable, and therefore agency telework participation and frequency  data will not likely be comparable with previous data calls because of  differences in definitions used, time periods of reporting, and individual  agency tracking methods. With the revised 2011 data call, OPM establishes a baseline it could use  to conduct a limited crosscheck of data collected through a  governmentwide automated telework data collection system, which OPM  plans to implement over the course of 2012 and 2013. OPM expects that  automated data collection will provide it increasingly more reliable data on  which to report progress. However, these efforts to improve future  automated data collection may result in changes to agencies’ methods of  data collection and a trade-off between the desire for consistency with  previous data calls for comparison purposes and a need to improve  overall data collection. Recommendations for Executive Action

To improve OPM’s annual reporting of telework to Congress, we  recommend that the OPM Director take the following two actions:
Ensure that the reliability limitations related to the 2011 telework data  call are clearly reported in its June 2012 report to Congress by fully  describing how existing measures of telework participation vary widely  in validity and reliability and limit the capability of OPM to reliably  report the actual level and frequency of telework participation. Continue efforts to improve data collection and gather information that  allows for the appropriate qualification of year-to-year comparisons  and informs users about the effects of data collection changes going  forward. In summary, OPM partially  concurred with our first recommendation and fully concurred with the  second. OPM also  provided a number of technical comments, which we incorporated as  appropriate. Appendix I: Objectives, Scope, and Methodology

This report (1) describes the Office of Personnel Management’s (OPM)  actions to respond to the requirements of the Telework Enhancement Act  of 2010 and (2) assesses how OPM is handling and addressing identified  data reliability issues in the 2011 telework data call. This included discussions on the  role of the Interagency Telework Measurement Group (ITMG), the  process for developing the definitions and key terms used in the data call,  the training and assistance provided to executive agency officials  responsible for completing the data call, and agency plans to address  outstanding data reliability issues associated with the data call.
Why GAO Did This Study

The Telework Enhancement Act of 2010 requires OPM to report to Congress on the degree of telework participation at executive agencies. To meet this requirement, OPM collects information on agency telework programs through an annual survey, which it refers to as the data call. However, concerns exist about the reliability of these data.
GAO was asked to assess OPMs: (1) actions in response to the requirements of the act and (2) handling of identified data reliability issues in the 2011 telework data call. To address these objectives GAO reviewed its previous reports addressing telework data reliability, and used the Office of Management and Budgets guidance for federal surveys to review OPMs (1) plans to collect telework participation data from agencies and (2) development of a data collection instrument. GAO interviewed key OPM officials about its implementation of the 2011 data call.

What GAO Found

To prepare for its reporting obligations under the Telework Enhancement Act of 2010, the Office of Personnel Management (OPM) assembled the Interagency Telework Measurement Group, consisting of officials from several federal agencies, to assist in revising the telework data callthe survey OPM has used since 2002 to collect telework data from executive agencies. This group standardized key terms such as telework, employee, and eligibility to promote a common reporting methodology among the agencies. The revised telework data call also included changes to the time period for which OPM requested agencies report telework data, and included more extensive training for respondents.
Because of changes made to the data call to allow OPM to meet requirements of the act and assist agencies in responding to the data call, OPM officials believe they will be able to provide to Congress an improved report on telework in June 2012. However, these changes also mean that OPM officials will not be able to use participation and frequency data from the 2011 data call to compare to data from previous years and across agencies. OPM officials have noted that this could limit OPMs ability to report agency progress in its first report to Congress. The ability to compare with previous years is affected by:
agencies use of methods of varying reliability to collect telework data, and some agencies made changes to their data collection systems for the 2011 data call. Executive agencies provide telework participation and frequency data by relying on estimates, counting telework agreements, or using automated time and attendance records.
modifications to the data call instrument, including changes to terminology and the time period during which telework data was requested. OPM officials said they expect these changes will improve the consistency of data. But if OPM reports progress based on data collected using changing terminology and from different time periods, the agency may reach erroneous conclusions.
Participants at the two data call training sessions may not have received the same reporting instructions, and uncertainty about whether all agency respondents attended training, created a risk that some respondents may be unaware of important terms and instructions. While some of the information provided at the two training sessions was similar, each session contained some new information, usually in response to questions raised at a previous session.
Future data call improvement efforts could result in a trade-off between the desire for maintenance of a consistent data series over time for comparison with previous data calls and a need to improve data collection. According to OPM, agencies will begin piloting automated telework data collection during 2012 and 2013. OPM expects that this method of data collection will provide it more reliable data than other methods. However, these efforts to standardize methods for tracking telework data may result in changes to agencies methods of data collection. The 2011 data call, notwithstanding its limitations, will be useful to help OPM identify and understand major changes in reported participation data that could occur during a transition to automated data collection.

What GAO Recommends

GAO recommends that OPM (1) clearly report reliability limitations with the 2011 telework data call in its June 2012 report to Congress and (2) continue efforts to improve data collection and gather information to allow for the appropriate qualification of year-to-year comparisons and inform users about the effects of data collection changes going forward. OPM partially concurred with the first recommendation. However, GAO believes it should report limitations in its annual report. OPM fully concurred with the second. OPM provided a number of technical comments which GAO incorporated as appropriate.