First, it criticized our use of readability tests to assess the clarity  of the six sample documents we reviewed. Second, CMS expressed concern about our examination of the usability of  the six sample documents. This question addressed which drug plan would  cost the least for a beneficiary with certain specified prescription drug  needs. Reading levels for the sample documents were challenging for at  least the 40 percent of seniors, who read at or below the 5th grade  level. Objective 1: Written Documents—Clarity (continued)

Undesirable features: The documents  used too much technical jargon,  often did not define difficult terms,  included sentences and some paragraphs that were too long,   did not use sufficient summaries to assist the reader in  identifying key points. Objective 1: Written Documents—Clarity (continued)

			Usability assessment: Beneficiaries and advisers to beneficiaries

were frustrated by the documents’ lack of clarity and often could not  complete the 18 assigned tasks. Objective 1: Written Documents—Clarity (continued)

			Some of the tasks that proved difficult included

computing projected total out-of-pocket costs for a plan that  provided Part D’s standard coverage (successfully completed by  none of the 11 beneficiaries and 2 of the 5 advisers),  evaluating whether it was possible to enroll in Medicare Part D  and keep drug coverage from a retiree health plan (successfully  completed by 2 beneficiaries and 2 advisers), and   determining the course of action for dual-eligibles who are  automatically enrolled in a plan that does not cover all drugs used  (successfully completed by 4 beneficiaries and 1 adviser). Objective 2: 1-800-MEDICARE Help Line Methodology (continued)

			To evaluate the accuracy and completeness of CSRs’ responses to

our five questions, we used three resources:  the prescription drug finder tool on the Medicare Web site,   the 1-800-MEDICARE scripts prepared by CMS and contractors  for CSRs to use in responding to callers’ questions, and    input from CMS officials on the criteria we used to evaluate  responses. In addition, it provided accurate—but incomplete— answers for about 3 percent of our calls. For example:

CSRs accurately and completely answered question 5 (whether a  beneficiary qualifies for extra help), which had a specific script, 90  percent of the time. Many of the responses were  inaccurate because the CSRs did not follow the available script or  provide sufficient information about the implications of the  beneficiary's decision. The rate is largely  caused by CSRs’ inappropriate responses—35 out of 100 times— that they were unable to answer the question without personal  identifying information, such as the beneficiary’s Medicare number  or date of birth. The prescription drug plan finder Web tool used by CSRs was not  operative at the time of our call (1 call). About 75 percent of calls were connected in less than 5 minutes. For calls where we waited more than 5 minutes to speak to a CSR, the wait time ranged from 5 minutes to over 55 minutes. The usability score was 47 percent for seniors and 53 percent for younger adults. While there is no widely accepted benchmark for usability, these scores indicate that using the site can be difficult. For example, tools such as the drug plan finder were complicated to use, and forms that collect information online from users were difficult to correct if the user made an error. Appendix II: Objectives, Scope, and Methodology

In this report, we assessed (1) the extent to which the Centers for  Medicare & Medicaid Services’ (CMS) written documents describe the  Medicare Part D prescription drug benefit in a clear, complete, and  accurate manner; (2) the effectiveness of CMS’s 1-800-MEDICARE help  line in providing accurate, complete, and prompt responses to callers  inquiring about the Part D benefit; (3) whether CMS’s Medicare Web site  presents information on the Part D benefit in a usable manner; and   (4) how CMS has used State Health Insurance Assistance Programs (SHIP)  to respond to the needs of Medicare beneficiaries for information on the  Part D benefit. To evaluate clarity, we contracted with the American Institutes for  Research (AIR)—a firm with experience in evaluating written material. To identify those important factors  associated with the Part D benefit, we reviewed relevant laws, regulations,  and 1-800-MEDICARE scripts prepared for customer service  representatives (CSR) to read to callers and obtained information from  advocacy groups. We evaluated the accuracy and  completeness of the responses by CSRs to the 500 calls by determining  whether key information was provided. This study consisted of three separate  evaluations. First, NN/g compared the site’s compliance with established  usability guidelines to determine a usability score to reflect the ease of  finding necessary information and performing various tasks.
Why GAO Did This Study

On January 1, 2006, Medicare began providing coverage for outpatient prescription drugs through its new Part D benefit. Beneficiaries who enroll in Part D may choose a drug plan from those offered by private plan sponsors under contract to the Centers for Medicare & Medicaid Services (CMS), which administers the Part D benefit. Beneficiaries have until May 15, 2006, to enroll in the Part D benefit and select a plan without the risk of penalties. GAO was asked to review the quality of CMS's communications on the Part D benefit. GAO examined 70 CMS publications to select 6 documents for review and contracted with the American Institutes for Research to evaluate the clarity of these texts; made 500 calls to the 1-800-MEDICARE help line; and contracted with the Nielsen Norman Group to evaluate the usability of the Medicare Web site.

What GAO Found

The information given in the six sample documents that GAO reviewed describing the Part D benefit was largely complete and accurate, although this information lacked clarity. The documents were unclear in two ways. First, although about 40 percent of seniors read at or below the fifth-grade level, the reading levels of these documents ranged from seventh grade to postcollege. Second, on average, the six documents did not comply with about half of 60 common guidelines for good communication. For example, the documents used too much technical jargon and often did not define difficult terms, such as formulary. Moreover, 16 beneficiaries and advisers that GAO tested reported frustration with the documents' lack of clarity and had difficulty completing the tasks assigned to them. Although the documents lacked clarity, they informed readers of enrollment steps and factors affecting coverage decisions and were consistent with laws, regulations, and agency guidance. Customer service representatives (CSR) responded to the 500 calls GAO placed to CMS's 1-800-MEDICARE help line accurately and completely about two-thirds of the time. Of the remainder, 18 percent of the calls received inaccurate responses, 8 percent of the responses were inappropriate given the question asked, and about 3 percent received incomplete responses. In addition, about 5 percent of GAO's calls were not answered, primarily because of disconnections. Accuracy and completeness rates of CSRs' responses varied significantly across the five questions GAO asked. For example, while CSRs provided accurate and complete responses to calls about beneficiaries' eligibility for extra help 90 percent of the time, the accuracy rate for calls concerning the drug plan that would cost the least for a specified beneficiary was 41 percent. For this question, the CSRs responded inappropriately for 35 percent of the calls by explaining that they could not identify the least costly plan without the beneficiary's personal information--even though CSRs had the information needed to answer the question. The time GAO callers waited to speak with CSRs also varied, ranging from no wait time to over 55 minutes. For 75 percent of the calls--374 of the 500--the wait was less than 5 minutes. The Part D benefit portion of the Medicare Web site can be difficult to use. GAO's test of the site's overall usability--the ease of finding needed information and performing various tasks--resulted in scores of 47 percent for seniors and 53 percent for younger adults, out of a possible 100 percent. While there is no widely accepted benchmark for usability, these scores indicate that using the site can be difficult. For example, the prescription drug plan finder was complicated to use and some of its key functions, such as "continue" and "choose a drug plan," were often not visible on the page without scrolling down.