1). Westat’s evaluation assessed youth self-reported drug use and  intermediate outcomes—such as youth and parent attitudes and beliefs  toward drug use and parental involvement with their children—that were  believed to influence youth drug use. In the conference report for fiscal year  2003 omnibus appropriations, the conferees reported that they were  “deeply disturbed by the lack of evidence that the National Youth Anti- Drug Media Campaign has had any appreciable impact on youth drug  use.” The conferees further acknowledged that while the evaluation  conducted under NIDA’s auspices showed “slight and sporadic impact on  the attitudes of parents, it has had no significant impact on youth  behavior.” The conferees further acknowledged that while other surveys  of youth drug use—such as Monitoring the Future, a survey of high school  youth—showed recent declines in drug use, “the NIDA study was  undertaken to measure the specific impact of the Media Campaign, not  simply to gauge general trends,” and the conferees stated that they “intend  to rely on the scientifically rigorous NIDA study to gauge the ultimate  impact of the campaign” and to reevaluate the use of taxpayer money to  support the campaign if the campaign continued to fail to demonstrate its  effectiveness. In 2002, the strategy for the campaign was redirected. The study period included  advertisements aired under the campaign’s Marijuana Initiative. Westat’s Evaluation Design, Use of Generally Accepted and Appropriate Sampling and Analytic Techniques, and Reliable Methods for Measuring Campaign Exposure Produced Credible Evidence to Support Its Findings

Westat was able to produce credible evidence to support its findings about  the relationship between exposure to campaign advertisements and both  drug use and intermediate outcomes by employing a longitudinal panel  design—i.e., collecting multiple observations on the same persons over  time—using generally accepted and appropriate sampling and analytic  techniques and establishing reliable and sufficiently powerful measures of  campaign exposure. In our view, Westat’s follow-up response rates resulted in a sample that  was sufficient to provide reliable findings about the effects of exposure on  outcomes. Westat’s Analytic Methods Aimed to Isolate Causal Effects of the Campaign and Did So Using Sophisticated Techniques That Enhanced the Strength of Its Findings

In it analysis, Westat used three types of evidence to draw inferences  about the effects of the campaign: (1) trend data—data that describe  increases or decreases in drug use and other outcomes over time;   (2) cross-section analysis—measures of association between exposure to  campaign messages and individual drug use beliefs, intentions, and  behaviors, at the time data were collected; and (3) longitudinal analysis— measures of association, for youth and parents who were observed at two  points in time, between exposure to campaign messages at the earlier time  on outcomes at the later time. On  the basis of the analysis of the relationship between exposure to campaign  advertisements and youth self-reported drug use in the NSPY data— assessments that used statistical methods to adjust for individual  differences and control for other factors that could explain changes in self- reported drug use—for the entire period covered by its evaluation, Westat  found no significant effects of exposure to the campaign on initiation of  marijuana by prior nonusing youth. Westat assessed the effects of two types  of exposure on initiation of marijuana use—general exposure and specific  exposure. Although ONDCP has pointed to declines in teen drug use and credited the  campaign along with other prevention efforts as contributing to significant  success in reducing teen drug use, trend data derived from the Monitoring  the Future survey that show declines in teen marijuana use from 2001 to  2005 do not explicitly take into account exposure to the campaign, and  therefore, by themselves, cannot be used as evidence of effectiveness. However, Westat’s  findings for the period from 2002 to 2004 showed that the campaign also  was not effective after ONDCP took these steps. We are sending copies of this report to other interested congressional  committees and the Director of the Office of National Drug Control Policy. Westat reported that it could not conclusively  rule out the first explanation for coverage losses. Westat’s sample had sufficient power to detect this  amount of annual change in youth attitudes.
Why GAO Did This Study

Between 1998 and 2004, Congress appropriated over $1.2 billion to the Office of National Drug Control Policy (ONDCP) for the National Youth Anti-Drug Media Campaign. The campaign aimed to prevent the initiation of or curtail the use of drugs among the nation's youth. In 2005, Westat, Inc., completed a multiyear national evaluation of the campaign. GAO has been mandated to review various aspects of the campaign, including Westat's evaluation which is the subject of this report. Applying generally accepted social science research standards, GAO assessed (1) how Westat provided credible support for its findings and Westat's findings about (2) attitudes, beliefs, and behaviors of youth and parents toward drug use and (3) youth self-reported drug use.

What GAO Found

GAO's review of Westat's evaluation reports and associated documentation leads to the conclusion that the evaluation provides credible evidence that the campaign was not effective in reducing youth drug use, either during the entire period of the campaign or during the period from 2002 to 2004 when the campaign was redirected and focused on marijuana use. By collecting longitudinal data--i.e., multiple observations on the same persons over time--using generally accepted and appropriate sampling and analytic techniques, and establishing reliable methods for measuring campaign exposure, Westat was able to produce credible evidence to support its findings about the relationship between exposure to campaign advertisements and both drug use and intermediate outcomes. In particular, Westat was able to demonstrate that its sample was not biased despite sample coverage losses, maintained high follow-up response rates of sampled individuals to provide for robust longitudinal analysis, established measures of exposure that could detect changes in outcomes on the order of magnitude that ONDCP expected for the campaign and that could reliably measure outcomes, and used sophisticated statistical methods to isolate causal effects of the campaign. Westat's findings on the effects of exposure on intermediate outcomes--theorized precursors of drug use--were mixed. Specifically, although sampled youth and parents' recall of campaign advertisements increased over time, they had good impressions of the advertisements, and they could identify the specific campaign messages, exposure to the advertisements generally did not lead youth to disapprove of using drugs and may have promoted perceptions among exposed youth that others' drug use was normal. Parents' exposure to the campaign led to changes in beliefs about talking about drug use with their children and the extent to which they had these conversations with their children. However, exposure did not appear to lead to increased monitoring of youth. Moreover, the evaluation was unable to demonstrate that changes in parental attitudes led to changes in youth attitudes or behaviors toward drug use. Westat's evaluation indicates that exposure to the campaign did not prevent initiation of marijuana use and had no effect on curtailing current users' marijuana use, despite youth recall of and favorable assessments of advertisements. Although general trend data derived from the Monitoring the Future survey and the Westat study show declines in the percentage of youth reportedly using marijuana from 2002 to 2004, the trend data do not explicitly take into account exposure to the campaign, and therefore, by themselves, cannot be used as evidence of effectiveness. In Westat's evaluation of relationships between exposure and marijuana initiation the only significant finding was of small unfavorable effects of the campaign exposure on marijuana initiation during some periods of data collection and in some subgroups.