Title: The reliability of a nutritional meta-analysis study,
Abstract: Background: Many researchers have studied the relationship between diet and
health. There are papers showing an association between the consumption of
sugar-sweetened beverages and Type 2 diabetes. Many meta-analyses use
individual studies that do not adjust for multiple testing or multiple modeling
and thus provide biased estimates of effect. Hence the claims reported in a
meta-analysis paper may be unreliable if the primary papers do not ensure
unbiased estimates of effect. Objective: Determine the statistical reliability
of 10 papers and indirectly the reliability of the meta-analysis study. Method:
Ten primary papers used in a meta-analysis paper and counted the numbers of
outcomes, predictors, and covariates. We estimated the size of the potential
analysis search space available to the authors of these papers; i.e. the number
of comparisons and models available. Since we noticed that there were
differences between predictors and covariates cited in the abstract and in the
text, we applied this formula to information found in the abstracts, Space A,
as well as the text, Space T, of each primary paper. Results: The median and
range of the number of comparisons possible across the primary papers are 6.5
and (2-12,288) for abstracts, and 196,608 and (3,072-117,117,952) the texts.
Note that the median of 6.5 for Space A is misleading as each primary study has
60-165 foods not mentioned in the abstract. Conclusion: Given that testing is
at the 0.05 level and the number of comparisons is very large, nominal
statistical significance is very weak support for a claim. The claims in these
papers are not statistically supported and hence are unreliable. Thus, the
claims of the meta-analysis paper lack evidentiary confirmation.