anchor.diff:        <li>Studies involving a drug or device that is manufactured in the United States are more likely to be registered than studies involving a drug or device manufactured outside of the United States.</li>
         <li>Studies subject to an IND or IDE are more likely to be registered (i.e., if the study is intended to support approval for marketing in the United States).</li>
         <li>Phase 1 adult drug studies or small feasibility studies of devices are less likely to be registered.</li>
  -      Studies in pediatric populations may be more likely to be registered.
  +      <li>Studies in pediatric populations may be more likely to be registered.</li>
       </ul>
   
  -    <h2>Duplicate records</h2>
  +    <h2>How are unique studies identified in AACT?</h2>
   
  -    <p>Studies registered at ClinicalTrials.gov are identified by a unique identifier, the NCT_ID. Because of the quality assurance measures applied by ClinicalTrials.gov staff on registration entries, we can be reasonably certain that each study (i.e., NCT_ID) entered in ClinialTrials.gov refers to a unique clinical study. However there may be a small number of duplicate records within the database [6].</p>
  +    <p>Studies registered at ClinicalTrials.gov are identified by a unique identifier, the NCT_ID.  Because of the quality assurance measures applied by ClinicalTrials.gov staff on registration entries, we can be reasonably certain that each study (i.e., NCT_ID) entered in ClinialTrials.gov refers to a unique clinical study, however a small number of duplicate records may exist in the database [6].  xxxxxxxx</p>
   
  -    <h2>Types of questions that can be investigated using the ClinicalTrials.gov data</h2>
  +    <h2>What types of questions can be investigated using ClinicalTrials.gov data?</h2>
   
  -    <p>The version of the ClinicalTrials.gov database that has been made publicly available through the CTTI and the DCRI contains study registration and results records. The registration records describe the study characteristics, including sponsor, disease condition, type of intervention, participant eligibility, anticipated enrollment, study design, locations, and outcome measures. Summary results and adverse events data are included in the current version of the AACT database. The article by Tse et al [8] may be helpful in understanding the components of the basic results that are reported at ClinicalTrials.gov.</p>
  +    <p>The AACT database contains both ‘study protocol’ and ‘results data’ elements. The protocol (or registration) records describe the study characteristics including sponsor, disease condition, type of intervention, participant eligibility, anticipated enrollment, study design, locations, and outcome measures. Summary results data elements including participant flow, baseline characteristics, outcome results, and frequencies of serious and other adverse events are included in AACT.  The article by Tse et al [8] xxxxx[link here] may be helpful in understanding the components of the basic results that are reported at ClinicalTrials.gov.</p>
   
  -    <h2>Using the study registration data</h2>
  +    <h3>Using the study registration data:</h3>
   
       <p>We anticipate that investigators will use the current database to explore the characteristics of selected subsets of clinical studies (e.g., typical enrollment for a phase 3 study in breast cancer patients), and to compare and contrast these characteristics across different subgroups of studies (e.g., sponsor; device versus drug intervention; or prevention versus treatment).</p>
   
  -    <h2>Using the study results and adverse events data</h2>
  +    <h2>Using the study results and adverse events data:</h2>
   
  -    <p>Researchers may be able to use the basic results and adverse events summary data reported at ClinicalTrials.gov for meta-analysis or systematic review (e.g., to compare the efficacy and safety of different types of diabetes therapies). However because only a small subset of studies registered in ClinicalTrials.gov are required to report results, the results data from ClinicalTrials.gov will most likely be a useful supplement to traditional data sources used for meta-analysis or systematic review, such as published and unpublished manuscripts and abstracts. Standard techniques for valid meta-analysis or systematic review (e.g., PRISMA statement [7]) should be used when determining how to appropriately identify and aggregate summary data gleaned from ClinicalTrials.gov and/or literature.</p>
  +    <p>Researchers may be able to use the basic results and adverse events summary data reported at ClinicalTrials.gov for meta-analysis or systematic review (e.g., to compare the efficacy and safety of different types of diabetes therapies).  However, because only a small subset of studies registered at ClinicalTrials.gov are required to report results, the results data from ClinicalTrials.gov will most likely be a useful supplement to traditional data sources used for a meta-analysis or systematic review, such as published and unpublished manuscripts and abstracts, rather than the core data source. Standard techniques for valid meta-analysis or systematic review (e.g., PRISMA statement xxxxxx  [7]) should be used when determining how to appropriately identify and aggregate summary data gleaned from ClinicalTrials.gov and/or literature.</p>
   
  -    <h2>Interpretation of variables</h2>
  +    <h2>How should data elements be interpreted?</h2>
   
  -    <p>When interpreting the study characteristics collected for a study registered with ClinicalTrials.gov, investigators are encouraged to refer to 2 NLM sites that provide authoritative definitions for the data element: study registration data elements and basic results data elements. Interpretation of a variable may depend on:</p>
  +    <p>When interpreting the data elements of a ClinicalTrials.gov study, you’re encouraged to refer to the authoritative definitions for data elements provided by the National Library of Medicine (NLM): xxxxxxxstudy registration data elements and basic results data elements.  Attribute interpretation may depend on:</p>
   
       <ul>
  -      <li>How the question was phrased. For example, the definition of “Sponsor” does not necessarily imply that the sponsor is the agency paying for the clinical study, as might be expected from the common use of the term.</li>
  -      <li>Whether the respondent can enter a free-text answer to a specific question, or is restricted to a fixed set of possible responses. **Note that the definition of a data element and the available responses may have changed over time. The most recent data element definitions are available at ClinicalTrials.gov Protocol Data Element Definitions Study Data Site and Results Data Site. A history of changes through September 2011 for the study variable definitions can be viewed in the Comprehensive Data Dictionary 2011 available <a href="http://ctti-clinicaltrials.org/files/documents/AACTcomprehensiveDataDictionaryV3_2011.xlsx" download>here</a>.</li>
  -      <li>Whether there is dependence between fields. Certain data elements need to be interpreted together with other data elements. For example, data elements such as enrollment date and completion date have a companion data element that indicates whether the value in the first field is an anticipated or actual value.</li>
  +      <li><b>How the question was phrased.</b> For example, the definition of “Sponsor” does not necessarily imply that the sponsor is the agency paying for the clinical study, as might be expected from the common use of the term.</li>
  +      <li><b>Whether the respondent can enter a free-text answer to a specific question, or is restricted to a fixed set of possible responses.</b> **Note that the definition of a data element and the available responses may have changed over time. The most recent data element definitions are available at ClinicalTrials.gov Protocol Data Element Definitions Study Data Site and Results Data Site. A history of changes through September 2011 for the study variable definitions can be viewed in the Comprehensive Data Dictionary 2011 available <a href="http://ctti-clinicaltrials.org/files/documents/AACTcomprehensiveDataDictionaryV3_2011.xlsx" download>here</a>.</li>
  +      <li><b>Whether there is dependence between fields.</b> Certain data elements need to be interpreted together with other data elements. For example, data elements such as enrollment date and completion date have a companion data element that indicates whether the value in the first field is an anticipated or actual value.</li>
       </ul>
   
       <p class="note">Note that the study record may be updated by the owner of the record at any time. Fields such as enrollment type may be changed from anticipated to actual, indicating that the value entered now reflects the actual rather than the planned enrollment. When data are downloaded, the result is a snapshot of the database at that particular time point, and the history of changes made to the field is lost.</p>
   
  -    <h2>Data completeness and accuracy</h2>
  +    <h2>How complete and accurate are the data?</h2>
   
       <p>The presence of a record in a dataset indicates that information was submitted to ClinicalTrials.gov for at least one variable in that data set before the data were downloaded from ClinicalTrials.gov. Some data elements are more/less likely than others to have missing information, depending on several known factors. For example:</p>
   
       <ul>
  -      <li>The data element being required by the FDAAA and/or the ClinicalTrials.gov website. Refer to data element definitions and the comprehensive data dictionary for specifics regarding these requirements. Requirements may have changed over the history of the ClincalTrials.gov database.</li>
  -      <li>The date when the data element was introduced. Not all data elements were included in the database at the time of its launch in 2000, but were added later. Studies registered after FDAAA when into effect must meet more requirements than studies registered earlier in the life of ClinicalTrials.gov.</li>
  -      <li>The branching structure of questions. The availability of certain questions to the person submitting data depends on answers to previous questions. For example, questions about bio-specimen retention are only available for observational studies. Therefore, interventional studies should be excluded when analyzing data elements pertaining to bio-specimens.</li>
  -      <li>The list of possible answers for data elements with a fixed set of responses. For example, questions that include “N/A” as a possible response are likely to have fewer missing values than questions that do not provide a “N/A” response.</li>
  +      <li><b>The data element being required by the FDAAA and/or the ClinicalTrials.gov website.</b> Refer to data element definitions xxxxxxxxxxxfor specifics regarding these requirements. Requirements may have changed over the history of the ClincalTrials.gov database.</li>
  +      <li><b>The date when the data element was introduced.</b> Not all data elements were included in the database at the time of its launch in 2000, but were added later. Studies registered after FDAAA when into effect must meet more requirements than studies registered earlier in the life of ClinicalTrials.gov.</li>
  +      <li><b>The branching structure of questions.</b> The availability of certain questions to the person submitting data depends on answers to previous questions. For example, questions about bio-specimen retention are only available for observational studies. Therefore, interventional studies should be excluded when analyzing data elements pertaining to bio-specimens.</li>
  +      <li><b>The list of possible answers for data elements with a fixed set of responses.</b> For example, questions that include “N/A” as a possible response are likely to have fewer missing values than questions that do not provide a “N/A” response.</li>
       </ul>
   
  -    <p>“Missingness” of data may also depend on other unknown factors. However, regardless of the cause of missing data, users of ClinicalTrials.gov datasets are encouraged to specify clearly how missing values and “N/A” values are handled in their analysis. For example, are studies with missing values excluded from statistics summarizing that data element, or are they included? In some cases, missing values may be imputed based on other fields (e.g., if a study has a single arm, it cannot employ a randomized design). In other cases, a sensitivity analysis may be appropriate for exploring the effect of different assumptions about the missing values on analysis results.</p>
  +    <p>“Missingness” of data may also depend on other unknown factors. However, regardless of the cause of missing data, users of ClinicalTrials.gov data sets are encouraged to specify clearly how missing values and “N/A” values are handled in their statistical analysis. For example, are studies with missing values excluded from statistics summarizing that data element, or are they included? In some cases, missing values may be imputed based on other fields (e.g., if a study has a single arm, it cannot employ a randomized design).</p>
   
       <p>Although the FDAAA and other requirements do not apply to all fields in the database, users might consider including only studies registered post-FDAAA (September 2007), or studies with a primary completion date after December 2007. This will help to limit the number of missing values across many data elements. Users could also consider annotating data elements used in analysis according to whether or not they are FDAAA-required fields, if the user believes this might affect the extent of missing data.</p>
   
  -    <p>Even when the data elements for a particular study are complete, users are cautioned to have modest expectations about the accuracy of the data. In particular, results data posted at ClinicalTrials.gov may not be subject to the same level of critical scrutiny as results published in a peer-reviewed journal. As described by Zarin and colleagues [1], ClinicalTrials.gov has implemented several measures to assure data quality. For example, staff applies automated business rules that alert providers when required data are missing or are internally inconsistent. In addition, some manual review is performed, and a record may be returned to the data provider if revision is required. However, ClinicalTrials.gov staff cannot always validate the accuracy of submitted data (e.g., against an independent source). As Zarin et al. note, “… individual record review has inherent limitations, and posting does not guarantee that the record is fully compliant with either ClinialTrials.gov or legal requirements” [1].</p>
  +    <p>Even when the data elements for a particular study are complete, users are cautioned to have modest expectations about their accuracy. In particular, results data posted at ClinicalTrials.gov may not be subject to the same level of critical scrutiny as results published in a peer-reviewed journal. As described by Zarin and colleagues [1]xxxxx, ClinicalTrials.gov has implemented several measures to ensure data quality. For example, NLM staff apply automated business rules that alert data-providers when required elements are missing or inconsistent. In addition, some manual review is performed by NLM, and a record may be returned to the data-provider if revision is required.  However, ClinicalTrials.gov staff cannot always validate the accuracy of submitted data (e.g., against an independent source).  As Zarin et al. note, “… individual record review has inherent limitations, and posting does not guarantee that the record is fully compliant with either ClinialTrials.gov or legal requirements” [1]xxxxx.</p>
   
  -    <p>During our own analysis of the ClinicalTrials.gov database, several potentially unrealistic values for numeric data elements were encountered, such as an anticipated enrollment of several million subjects. When aggregate summaries of numeric data are provided, analysts are encouraged to use measures that are robust to outliers, such as medians and interquartile ranges, rather than measures such as means ± SD, which could be strongly influenced by unusually large or small values. Users may also wish to run their own consistency checks (e.g., to compare whether the number of arm descriptions provided for the study matches the data element that quantifies the number of arms in the study design).</p>
  +    <p>During our own analysis of the ClinicalTrials.gov database, several extreme values for numeric data elements were encountered, such as an anticipated enrollment of several million subjects. Before proceeding with aggregate analysis, users are encouraged to review data distributions in order to select appropriate analysis methods, and to run their own consistency checks (e.g., to compare whether the number of arm descriptions provided for the study matches the data element that quantifies the number of arms in the study design) as needed.</p>
   
  -    <h2>AACT is a snapshot at one time point</h2>
  +    <h2>Downloaded AACT is a snapshot at one point in time</h2>
   
  -    <p>The data downloaded from ClinicalTrials.gov and stored in the database for Aggregate Analysis of ClinicalTrials.gov (AACT) is a snapshot of the information that was publicly available at ClinicalTrials.gov on the download date. Data submitters may update their ClinicalTrials.gov study record at any time but a particular version of AACT only captures the information present at one point in time. Although changes to a study are stored in an archive history at ClinicalTrials.gov, these changes are not captured in a particular version of AACT. A user may find that information contained in AACT differs from information currently listed on ClinicalTrials.gov.</p>
  +    <p>The data downloaded from ClinicalTrials.gov and stored in AACT is a snapshot of the information that was publicly available at ClinicalTrials.gov on the download date.  Data submitters may update their ClinicalTrials.gov study record at any time but a particular instance of AACT only captures the information present at one point in time.  Although changes to a study are stored in an archive history at ClinicalTrials.gov, these changes are not currently captured in a particular version of AACT. </p>
  +
  +<h3>Should we alter this to reflect how the person may access AACT? If they download it is a snapshot on that date. If the access directly - it is up to date as of the previous day? Or should it be removed and a blurb be placed somewhere else on the AACT2 site?</h3>
   
   
       <h2>Use of appropriate statistical inference</h2>
   
  -    <p>If the AACT results data are to be used to support a meta-analysis or systematic review of the safety or efficacy of a particular intervention, then standard methods of meta-analysis or systematic review (e.g., the PRISMA statement [7]) should be used to appropriately account for study-to-study variability and other sources of uncertainty or bias. We recommend that authors consider the following points when deciding whether to report p-values, confidence intervals, or other probability-based inference when performing aggregate analysis of the ClinicalTrials.gov database.</p>
  +    <p>If the AACT results data are to be used to support a meta-analysis or systematic review of the safety or efficacy of a particular intervention, then standard methods of meta-analysis or systematic review (e.g., the PRISMA statementxxxxxx[7]) should be used to appropriately account for study-to-study variability and other sources of uncertainty or bias. We recommend that authors consider the following points when deciding whether to report p-values, confidence intervals, or other probability-based inference when performing aggregate analysis of the ClinicalTrials.gov database:  </p>
   
       <h3>Is the data-generating mechanism random?</h3>
       <p>Methods of statistical inference such as p-values and 95% confidence intervals are most appropriate when used to quantify the uncertainty of estimates or comparisons due to a random process that generates the data. Examples of such processes include selection of a random sample of subjects from a broader population, randomly assigning a treatment to a cohort of subjects, or a coin toss about which we aim to predict future results.</p>
  
positive.code: <section class="pointsToConsider">
    <div class="wrap">
  
      <h2>Introduction</h2>
  
      <p>  This document suggests points for investigators to consider when planning a statistical analysis of the database for Aggregate Analysis of ClinicalTrials.gov (AACT). It is not intended to be a comprehensive guide for using or analyzing data in AACT. AACT includes both the study registration fields and the basic results and adverse events reporting fields.</p>
  
      <p> <a href="http://ec2-52-87-233-209.compute-1.amazonaws.com/aact_points_to_consider.docx" download>Download a word document</a> containing the   information on this page.</p>
  
      <h2>Population: studies likely to be represented</h2>
  
      <p>Virtually any clinical study may be registered at ClinicalTrials.gov and therefore be included in AACT. However, the registry is more likely to include certain types of study relative to others. These biases are summarized by Zarin et al. [1]:</p>
  
      <p>“… [T]here are undoubtedly trials that are not registered in ClinicalTrials.gov or any other publicly accessible registry. Coverage in ClinicalTrials.gov is likely to be most complete for trials of drugs or devices that are sponsored by U.S.-based or multinational organizations (e.g., major pharmaceutical companies).”</p>
  
      <p>The ClinicalTrials.gov trial registry was released for the registration of studies on February 29, 2000. The registration of studies and reporting of results and adverse events has been mandated to a large extent by requirements (both legal and institutional) implemented as part of the Food and Drug Administration Amendments Act (FDAAA), as well as by requirements introduced by the International Committee of Medical Journal Editors (ICMJE) and the European Medicines Agency (EMA) regarding registration of clinical studies. Table 1 describes the scope of these requirements.</p>
  
      <p>Based on these requirements, the following are examples of characteristics that may influence the likelihood that a study is included in the ClinicalTrials.gov registry:</p>
  
  
negative.code: <div class="wrap">
  
    <div class="basicHero">
      <h1>Guide to Using Aggregate Analysis of ClinicalTrials.gov (AACT) for Statistical and Aggregate Analysis</h1>
  
    </div>
  
  