Reported methodologic quality and discrepancies between large and small randomized trials in meta-analyses. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. Association of funding and conclusions in randomized drug trials: a reflection of treatment effect or adverse events? Assessment of risk of bias among pediatric randomized controlled trials. Randomised controlled trial of effects of coordinating care for terminally ill cancer patients. Secular changes in the quality of published randomized clinical trials in rheumatology. Treatment success in pragmatic randomised controlled trials: a review of trials funded by the UK Health Technology Assessment programme. Randomized and non-randomized patients in clinical trials: experiences with comprehensive cohort studies. Undisclosed changes in outcomes in randomized controlled trials: an observational study. The uncertainty principle and industry-sponsored research. An analysis of the effect of funding source in randomized clinical trials of second generation antipsychotics for the treatment of schizophrenia. A randomised trial of the effects of an additional communication strategy on recruitment into a large-scale, multi-centre trial. Gynaecologic surgery from uncertainty to science: evolution of randomized control trials. Impact of random assignment on study outcome: an empirical examination. Quality of randomised trials in COPD.
To explore whether reported methodologic quality affects estimated intervention effects in randomized trials and contributes to discrepancies between the results of large randomized trials and small randomized trials in meta-analyses.
                Meta-analyses of randomized trials that included at least one large trial (>/=1000 participants) were included, regardless of the therapeutic area. Eligible meta-analyses were identified through electronic searches and bibliographies of relevant articles.
                Full-length randomized trials.
                Methodologic quality was assessed according to reported randomization, double blinding, and follow-up as separate components and by using the Jadad composite scale.
                Fourteen meta-analyses involving 190 randomized trials from eight therapeutic areas were included. Compared with large trials, intervention effects were exaggerated in small trials with inadequate allocation sequence generation (ratio of odds ratios, 0.46 [95% CI, 0.25 to 0.83]; P = 0.011), inadequate allocation concealment (ratio of odds ratios, 0.49 [CI, 0.27 to 0.86]; P = 0.014), and no double blinding (ratio of odds ratios, 0.52 [CI, 0.28 to 0.96]; P = 0.01). Large trials did not differ significantly from small trials with adequate generation of the allocation sequence, adequate allocation concealment, or adequate double blinding. No association was seen between reported follow-up and intervention effects. The Jadad scale provided no additional information because the scale and the quality components overlapped substantially.
                Inadequate generation of the allocation sequence, allocation concealment, and double blinding lead to exaggerated estimates of intervention benefit and may contribute to discrepancies between the results of large randomized trials and small randomized trials in meta-analyses. To determine if inadequate approaches to randomized controlled trial design and execution are associated with evidence of bias in estimating treatment effects.
                An observational study in which we assessed the methodological quality of 250 controlled trials from 33 meta-analyses and then analyzed, using multiple logistic regression models, the associations between those assessments and estimated treatment effects.
                Meta-analyses from the Cochrane Pregnancy and Childbirth Database.
                The associations between estimates of treatment effects and inadequate allocation concealment, exclusions after randomization, and lack of double-blinding.
                Compared with trials in which authors reported adequately concealed treatment allocation, trials in which concealment was either inadequate or unclear (did not report or incompletely reported a concealment approach) yielded larger estimates of treatment effects (P < .001). Odds ratios were exaggerated by 41% for inadequately concealed trials and by 30% for unclearly concealed trials (adjusted for other aspects of quality). Trials in which participants had been excluded after randomization did not yield larger estimates of effects, but that lack of association may be due to incomplete reporting. Trials that were not double-blind also yielded larger estimates of effects (P = .01), with odds ratios being exaggerated by 17%.
                This study provides empirical evidence that inadequate methodological approaches in controlled trials, particularly those representing poor allocation concealment, are associated with bias. Readers of trial reports should be wary of these pitfalls, and investigators must improve their design, execution, and reporting of trials. Selective reporting of outcomes within published studies based on the nature or direction of their results has been widely suspected, but direct evidence of such bias is currently limited to case reports.
                To study empirically the extent and nature of outcome reporting bias in a cohort of randomized trials.
                Cohort study using protocols and published reports of randomized trials approved by the Scientific-Ethical Committees for Copenhagen and Frederiksberg, Denmark, in 1994-1995. The number and characteristics of reported and unreported trial outcomes were recorded from protocols, journal articles, and a survey of trialists. An outcome was considered incompletely reported if insufficient data were presented in the published articles for meta-analysis. Odds ratios relating the completeness of outcome reporting to statistical significance were calculated for each trial and then pooled to provide an overall estimate of bias. Protocols and published articles were also compared to identify discrepancies in primary outcomes.
                Completeness of reporting of efficacy and harm outcomes and of statistically significant vs nonsignificant outcomes; consistency between primary outcomes defined in the most recent protocols and those defined in published articles.
                One hundred two trials with 122 published journal articles and 3736 outcomes were identified. Overall, 50% of efficacy and 65% of harm outcomes per trial were incompletely reported. Statistically significant outcomes had a higher odds of being fully reported compared with nonsignificant outcomes for both efficacy (pooled odds ratio, 2.4; 95% confidence interval [CI], 1.4-4.0) and harm (pooled odds ratio, 4.7; 95% CI, 1.8-12.0) data. In comparing published articles with protocols, 62% of trials had at least 1 primary outcome that was changed, introduced, or omitted. Eighty-six percent of survey responders (42/49) denied the existence of unreported outcomes despite clear evidence to the contrary.
                The reporting of trial outcomes is not only frequently incomplete but also biased and inconsistent with protocols. Published articles, as well as reviews that incorporate them, may therefore be unreliable and overestimate the benefits of an intervention. To ensure transparency, planned trials should be registered and protocols should be made publicly available prior to trial completion. Previous studies indicate that industry-sponsored trials tend to draw proindustry conclusions.
                To explore whether the association between funding and conclusions in randomized drug trials reflects treatment effects or adverse events.
                Observational study of 370 randomized drug trials included in meta-analyses from Cochrane reviews selected from the Cochrane Library, May 2001. From a random sample of 167 Cochrane reviews, 25 contained eligible meta-analyses (assessed a binary outcome; pooled at least 5 full-paper trials of which at least 1 reported adequate and 1 reported inadequate allocation concealment). The primary binary outcome from each meta-analysis was considered the primary outcome for all trials included in each meta-analysis. The association between funding and conclusions was analyzed by logistic regression with adjustment for treatment effect, adverse events, and additional confounding factors (methodological quality, control intervention, sample size, publication year, and place of publication).
                Conclusions in trials, classified into whether the experimental drug was recommended as the treatment of choice or not.
                The experimental drug was recommended as treatment of choice in 16% of trials funded by nonprofit organizations, 30% of trials not reporting funding, 35% of trials funded by both nonprofit and for-profit organizations, and 51% of trials funded by for-profit organizations (P<.001; chi2 test). Logistic regression analyses indicated that funding, treatment effect, and double blinding were the only significant predictors of conclusions. Adjusted analyses showed that trials funded by for-profit organizations were significantly more likely to recommend the experimental drug as treatment of choice (odds ratio, 5.3; 95% confidence interval, 2.0-14.4) compared with trials funded by nonprofit organizations. This association did not appear to reflect treatment effect or adverse events.
                Conclusions in trials funded by for-profit organizations may be more positive due to biased interpretation of trial results. Readers should carefully evaluate whether conclusions in randomized trials are supported by data. The goal was to assess the risk of bias among pediatric, randomized, controlled trials (RCTs) reported in 8 high-impact journals.
                We searched PubMed for all pediatric RCTs reported between July 1, 2007, and June 30, 2008, in 8 journals with high impact factors. Using Cochrane Collaboration methods for risk assessment, we evaluated all reports for risk of bias according to domain (ie, randomized sequence generation, allocation concealment, masking, incomplete outcome data, selective outcome reporting, and other). We used multiple logistic regression to test for associations between the presence of a high risk of bias according to domain and funding source, intervention type, trial registration, and multicenter status.
                Industry-funded RCTs were more likely to show a high risk of bias for sequence generation, compared with government-funded RCTs (adjusted odds ratio [aOR]: 6.1 [95% confidence interval [CI]: 1.70- 21.89]), and behavioral/educational trials were more likely to show a high risk of bias for sequence generation (aOR: 2.8 [95% CI: 1.06-7.36]) and allocation concealment (aOR: 4.09 [95% CI: 1.69-9.90]), compared with drug trials. Registered trials were less likely to have a high risk of bias for sequence generation, compared with nonregistered trials (aOR: 0.33 [95% CI: 0.15-0.71]).
                Overall, we found a large proportion of pediatric RCT reports with a high risk of bias for sequence generation and allocation concealment. Factors associated with a high risk of bias included industry funding and assessment of behavioral/educational interventions, whereas trial registration was associated with a lower risk of bias. To measure effects on terminally ill cancer patients and their families of coordinating the services available within the NHS and from local authorities and the voluntary sector.
                Randomised controlled trial.
                Inner London health district.
                Cancer patients were routinely notified from 1987 to 1990. 554 patients expected to survive less than one year entered the trial and were randomly allocated to a coordination or a control group.
                All patients received routinely available services. Coordination group patients received the assistance of two nurse coordinators, whose role was to ensure that patients received appropriate and well coordinated services, tailored to their individual needs and circumstances.
                Patients and carers were interviewed at home on entry to the trial and at intervals until death. Interviews after bereavement were also conducted. Outcome measures included the presence and severity of physical symptoms, psychiatric morbidity, use of and satisfaction with services, and carers' problems. Results from the baseline interview, the interview closest to death, and the interview after bereavement were analysed.
                Few differences between groups were significant. Coordination group patients were less likely to suffer from vomiting, were more likely to report effective treatment for it, and less likely to be concerned about having an itchy skin. Their carers were more likely to report that in the last week of life the patient had had a cough and had had effective treatment for constipation, and they were less likely to rate the patient's difficulty swallowing as severe or to report effective treatment for anxiety. Coordination group patients were more likely to have seen a chiropodist and their carers were more likely to contact a specialist nurse in a night time emergency. These carers were less likely to feel angry about the death of the patient.
                This coordinating service made little difference to patient or family outcomes, perhaps because the service did not have a budget with which it could obtain services or because the professional skills of the nurse-coordinators may have conflicted with the requirements of the coordinating role. To assess the quality of published randomized clinical trials (RCTs) in rheumatology and to determine whether there has been improvement in quality between 2 time periods, 1987-1988 and 1997-1998.
                Using MEDLINE and a hand search of selected rheumatology journals, we identified RCTs of adult rheumatic diseases published in English in 1987-1988 or 1997-1998. We examined trial quality with an expanded version of the Jadad scale, which assesses the adequacy of reported random sequence generation, allocation concealment, blinding, and analysis. All trials were read by 1 reviewer, with prior standardization using a random sample read by 2 reviewers. We also evaluated "high"- versus "low"-impact journals based on citation index.
                Two hundred forty RCTs (1987-1988 119 RCTs, 1997-1998 121 RCTs) were assessed. Results showed improvement in the quality of the trials, but the rates of reported random sequence generation, allocation concealment, power, and intent-to-treat analyses were persistently low. Low rates of reports of random sequence generation, allocation concealment, and intent-to-treat analyses were present even in the high-impact journals.
                There has been improvement in the quality of reporting of RCTs in rheumatology between 1987-1988 and 1997-1998. However, methodologic problems such as lack of allocation concealment, inadequate random sequence generation, lack of reporting of power, and lack of intent-to-treat analyses remain common. Many of these problems are established sources of bias in RCTs and are easily rectifiable. Previous research reviewed treatment success and whether the collective uncertainty principle is met in RCTs in the US National Cancer Institute portfolio. This paper classifies clinical trials funded by the UK HTA programme by results using the method applied to the US Cancer Institute trials, and compares the two portfolios.
                Data on all completed randomised controlled trials funded by the HTA programme 1993-2008 were extracted. Each trial's primary results was classified into six categories; 1) statistically significant in favour of the new treatment, 2) statistically significant in favour of the control treatment 3) true negative, 4) truly inconclusive, 5) inconclusive in favour of new treatment or 6) inconclusive in favour of control treatment. Trials were classified by comparing the 95% confidence interval for the difference in primary outcome to the difference specified in the sample size calculation. The results were compared with Djulbegovic's analysis of NCI trials.
                Data from 51 superiority trials were included, involving over 48,000 participants and a range of diseases and interventions. 85 primary comparisons were available because some trials had more than two randomised arms or had several primary outcomes. The new treatment had superior results (whether significant or not) in 61% of the comparisons (52/85 95% CI 49.9% to 71.6%). The results were conclusive in 46% of the comparisons (19% statistically significant in favour of the new treatment, 5% statistically significant in favour of the control and 22% true negative). The results were classified as truly inconclusive (i.e. failed to answer the question asked) for 24% of comparisons (20/85). HTA trials included fewer truly inconclusive and statistically significant results and more results rated as true negative than NCI trials.
                The pattern of results in HTA trials is similar to that of the National Cancer Institute portfolio. Differences that existed were plausible given the differences in the types of trials -HTA trials are more pragmatic. The results indicate HTA trials are compatible with equipoise. This classification usefully summarises the results from clinical trials and enables comparisons of different portfolios of trials. In clinical research, randomized trials are widely accepted as the definitive method of evaluating the efficacy of therapies. Random assignment of patients to treatment ensures internal validity of the comparison of new treatments with controls. An assessment of external validity can best be achieved by comparing the randomized study sample to the population of patients who met the eligibility criteria but did not consent to randomization. The Comprehensive Cohort Study (CCS) is designed to recruit all patients fulfilling the clinical eligibility criteria regardless of their consent to randomization. The CCS concept was adopted in the major clinical trials of the German Breast Cancer Study Group (GBSG) conducted between 1983 and 1989. In this period 124 centres recruited 2084 patients in three clinical trials. 734 (35 per cent) of these patients accepted being randomized, while 1350 (65 per cent) chose one of the treatments under study; the randomization rates differed remarkably between trials. In this paper we examine the representativeness of the randomized patients in the three trials. Based on a median follow-up of about 5 years we present results on the external validity of the treatment effects estimated in the randomized patients by means of Cox's proportional hazards model and compare them between trials. We discuss advantages and disadvantages of the CCS design and conclude that its use is only justified under extraordinary circumstances. We wanted to investigate the frequency of undisclosed changes in the outcomes of randomized controlled trials (RCTs) between trial registration and publication.
                Using a retrospective, nonrandom, cross-sectional study design, we investigated RCTs published in consecutive issues of 5 major medical journals during a 6-month period and their associated trials registry entries. Articles were excluded if they did not have an available trial registry entry, did not have analyzable outcomes, or were secondary publications. The primary outcome was the proportion of publications in which the primary outcome of the trial was, without disclosure, changed between that recorded in the trial registry and that reported in the final publication. The secondary outcome was the proportion of publications in which the secondary outcome was changed without disclosure.
                We reviewed 158 reports of RCTs and included 110 in the analysis. In 34 (31%), a primary outcome had been changed, and in 77 (70%), a secondary outcome had been changed.
                There are substantial and important undisclosed changes made to the outcomes of published RCTs between trial registration and publication. This finding has important implications for the interpretation of trial results. Disclosure and discussion of changes would improve transparency in the performance and reporting of trials. Reporting of pharmaceutical-industry-sponsored randomised clinical trials often result in biased findings, either due to selective reporting of studies with non-equivalent arms or publication of low-quality papers, wherein unfavourable results are incompletely described. A randomised trial should be conducted only if there is substantial uncertainty about the relative value of one treatment versus another. Studies in which intervention and control are thought to be non-equivalent violates the uncertainty principle.
                We examined the quality of 136 published randomised trials that focused on one disease category (multiple myeloma) and adherence to the uncertainty principle. To evaluate whether the uncertainty principle was upheld, we compared the number of studies favouring experimental treatments over standard ones. We analysed data according to the source of funding.
                Trials funded solely or in part by 35 profit-making organisations had a trend toward higher quality scores (mean 2.94 [SD 1.3]; median 3) than randomised trials supported by 95 governmental or other non-profit organisations (2.4 [0.8]; 2; p=0.06). Overall, the uncertainty principle was upheld, with 44% of randomised trials favouring standard treatments and 56% innovative treatments (p=0.17); mean and median preference evaluation scores were 3.7 (1.0) and 4. However, when the analysis was done according to the source of funding, studies funded by non-profit organisations maintained equipoise favouring new therapies over standard ones (47% vs 53%; p=0.608) to a greater extent than randomised trials supported solely or in part by profit-making organisations (74% vs 26%; p=0.004).
                The reported bias in research sponsored by the pharmaceutical industry may be a consequence of violations of the uncertainty principle. Sponsors of clinical trials should be encouraged to report all results and to choose appropriate comparative controls. The effect of funding source on the outcome of randomized controlled trials has been investigated in several medical disciplines; however, psychiatry has been largely excluded from such analyses. In this article, randomized controlled trials of second generation antipsychotics in schizophrenia are reviewed and analyzed with respect to funding source (industry vs. non-industry funding).
                A literature search was conducted for randomized, double-blind trials in which at least one of the tested treatments was a second generation antipsychotic. In each study, design quality and study outcome were assessed quantitatively according to rating scales. Mean quality and outcome scores were compared in the industry-funded studies and non-industry-funded studies. An analysis of the primary author's affiliation with industry was similarly performed.
                Results of industry-funded studies significantly favored second generation over first generation antipsychotics when compared to non-industry-funded studies. Non-industry-funded studies showed a trend toward higher quality than industry-funded studies; however, the difference between the two was not significant. Also, within the industry-funded studies, outcomes of trials involving first authors employed by industry sponsors demonstrated a trend toward second generation over first generation antipsychotics to a greater degree than did trials involving first authors employed outside the industry (p=0.05).
                While the retrospective design of the study limits the strength of the findings, the data suggest that industry bias may occur in randomized controlled trials in schizophrenia. There appears to be several sources by which bias may enter clinical research, including trial design, control of data analysis and multiplicity/redundancy of trials. Timely participant recruitment remains a significant challenge for most clinical trials. We evaluated the effects on participant recruitment of communication between the central trial coordinators and the clinical sites in the setting of a large international multi-centre clinical trial. The effects of communication were determined in a single-blind randomised controlled trial involving 167 clinical sites in 19 countries. Clinical sites were randomised to either additional or usual communication strategies - the additional communication group received a communication package based on additional, individually-tailored feedback about recruitment, in addition to the usual correspondence from the central trial coordinators that was provided to the control group. The two study outcomes were the median time to half randomisation target and the median total number of participants randomised per clinical site. Eighty-five clinical centres were randomised to receive additional communication and 82 to receive usual communication. At the conclusion of recruitment, there was no significant difference in the median number of participants randomised per centre between the additional and usual groups (37.5 vs. 37.0, p=0.68). The median time to half randomisation target was lower in the additional communication group compared to the usual group, however this difference did not achieve conventional levels of statistical significance (4.4 months vs. 5.8 months, p=0.08). The findings suggest that the additional communication strategy may be of some incremental benefit in helping sites achieve recruitment targets sooner. It is now accepted that both medical and surgical practice should be based on reliable and sound clinical evidence. However, randomized control trials comparing surgical interventions have been associated with many problems. The aim of this review is to assess if there has been progress made in establishing the evidence base for surgical interventions in gynaecology.
                Relevant reviews were identified from Cochrane Database of Systematic Reviews (Issue , 2006) and data from individual randomized control trials extracted. Chi-squared test was used to compare quality pre- and post-Consolidated Standards of Reporting Trials (CONSORT) statement. Meta-regression analyses were performed to test the hypothesis that effect size decreased over time. Further multiple linear regression analyses were used to test the hypothesis that precision increased over time and finally a logistic regression model was used to estimate whether treatment effects differed between trials with and without allocation concealment.
                Twenty-three relevant reviews were identified, including 94 trials. The proportion of studies reporting allocation concealment significantly increased after the introduction of the CONSORT statement (P = 0.002). There was a trend towards improvement in precision over time. Similarly, there was a reduction in size of treatment effect over time (log of the ratio of odds ratios per year 0.96; 95% confidence interval 0.93-0.99, P = 0.04).
                Gynaecologic surgical practice appears to be benefiting from improvement in its research base in a subject where practitioners do not participate readily in randomized evaluation. Sixty research investigations published in the biomedical literature were analyzed to examine the effect of design attributes on outcome. All 60 studies included a controlled trial involving a pretest, a therapeutic intervention, and a posttest across at least two groups. Thirty of the trials used random assignment of participants to treatment or control conditions and 30 trials employed some nonrandom method of subject assignment. Trial results were aggregated and evaluated by comparing effect sizes for the primary statistical test of the hypothesis. Data analysis revealed that the trial results, as measured by effect size, did not vary across therapeutic trials using random assignment versus those using nonrandom allocation of subjects. The impact of design attributes in the interpretation of multiple clinical trials addressing a similar research question is examined. The argument is made that various design attributes frequently associated with methodological quality should be considered as important moderator variables and their influence on trial outcome should not be assumed a priori but rather examined empirically. Randomised trials can provide high-level evidence to inform treatment decisions. Since their quality in respiratory medicine is largely unknown, we assessed the quality of a large set of chronic obstructive pulmonary disease (COPD) trials. As a marker of trial quality, we assessed the procedure and concealment of random allocation, and the conduct of an intention-to-treat-analysis in 344 randomised trials published between 1957 and 2006. We used ordered logistic regression to assess the association between trial quality and type of intervention, type of journal, journal impact factor and year of publication. 257 (75%) trials assessed pharmacological and 87 (25%) assessed nonpharmacological interventions. The generation of appropriate randomisation was reported in 27.0% of the trials, concealment of random allocation in 11.6% and an intention-to-treat analysis in 21.8% of trials. Significantly higher quality was found in trials on nonpharmacological interventions (OR 2.49, 95% CI 1.56-3.99), and in trials published in general medical journals (versus specialised journals; OR 2.25, 95% CI 1.30-3.90) and after 2000 (versus 1957-2000; OR 2.28, 95% CI 1.45-3.58). The association of quality with a high impact factor was of borderline significance (p = 0.06). The quality of many COPD trials is low but tends to become better since the adoption of the CONSORT (Consolidated Standards of Reporting Trials) statement.
The results of randomised and non-randomised studies sometimes differed. In some instances non-randomised studies yielded larger estimates of effect and in other instances randomised trials yielded larger estimates of effect. The results of controlled trials with adequate and inadequate/unclear concealment of allocation sometimes differed. When differences occurred, most often trials with inadequate or unclear allocation concealment yielded larger estimates of effects relative to controlled trials with adequate allocation concealment. However, it is not generally possible to predict the magnitude, or even the direction, of possible selection biases and consequent distortions of treatment effects from studies with non-random allocation or controlled trials with inadequate or unclear allocation concealment.