anchor.diff:  
       <p>c) Determine appropriate formulas for quantifying differences between groups or summarizing population variability.  This quantification could take into account of the observed difference, variability in the data, and the number of observations.  Examples are provided:</p>
   
  -    <ul>
  +    <ul class='pointsToConsider'>
         <li>When summarizing a continuous characteristic such as enrollment, the analyst might choose to report the median and 5th to 95th percentiles.</li>
         <li>To quantify signal to noise, the analyst could calculate a t-statistic or a chi-squared statistic (without the p-value) and rank differences between 2 groups based on these values.  The analyst might pre-specify a threshold (e.g., absolute value of 3) to flag notable differences.</li>
       </ul>
   
       <h2>Specific tips for working with the AACT database</h2>
   
  -    <ul>
  +    <ul class='pointsToConsider'>
         <li>Users are encouraged to use the Schema Diagram xxxxx to determine relationships between different AACT tables/datasets (referred to as ‘datasets’ on this page). These relationships determine how datasets may be linked or merged together in software such as SAS & SQL.</li>
   
         <li>The NCT_ID uniquely identifies each study; it serves as the primary key in the STUDIES dataset.  In other words, each record in the STUDIES dataset has a unique NCT_ID value.  NCT_ID also appears in every dataset related to STUDIES so that every record in every dataset can link back to the study to which it refers.</li>
   
         <li>Every dataset other than STUDIES has a primary key named ‘id’, which provides an integer that uniquely identifies each row in that dataset. (The STUDIES dataset uses ‘NCT_ID’ instead of ‘id’  as the unique identifier for each row.)</li>
  -			<li>To link dataset information to the study to which it refers, you simply match the NCT_IDs.  For example, every record in the CONDITIONS dataset with an NCT_ID of ‘NCT0000001’ refers to the study with the NCT_ID: ‘NCT0000001’ (saved in STUDIES.nct_id).  The CONDITIONS dataset may contain multiple records with an NCT_ID of  ‘NCT0000001’ which means this study was defined in ClinicalTrials.gov as being associated with the conditions listed in CONDITIONS with that NCT_ID.</li>
  -			<li>Information in several datasets are also related to information in other datasets.  In this case, the dataset that belongs to another dataset will include a foreign key that identifies the record to which it belongs. Foreign keys are always named according to a simple rule:  the singular name of the related dataset followed by: ‘_id’.   For example, FACILITY_CONTACTS includes a data element: facility_id which is the foreign key to FACILITY.id</li>
  -			<li> Each record’s foreign_key (ie. facility_id) contains the value of the unique identifier (id) of the record in the other dataset to which it belongs. For example, a facility may have multiple contacts.  To find the contacts for a particular facility, look for the records in FACILITY_CONTACTS where the value in facility_id is same as the value in id for that facility in FACILITIES.  In short, datasets are related to each other with this pattern: dataset1.<dataset2_name>_id = dataset2.id </li>
  -
  -    </ul>
  -		<p>Example:</p>
  -		<p>facilities.id = contacts.facility_id  (this finds  all contacts for a facility)</p>
  -		<p>or inversely</p>
  -		<p>contacts.facility_id = facilities.id   (this finds the facility for a particular contact)</p>
  -
  -		<h2>Information about trial sites (FACILITIES and COUNTRIES) </h2>
  -		<p> Information about organizations (trial sites) where the study is/was conducted is stored in the FACILITIES dataset.  This is a snapshot of the facility information that was included in the study record on the date that information was downloaded from ClinicalTrials.gov.  Naturally, study sites initiated after ClinicalTrials.gov data were downloaded to AACT are not included in AACT.</p>
  +      <li>To link dataset information to the study to which it refers, you simply match the NCT_IDs.  For example, every record in the CONDITIONS dataset with an NCT_ID of ‘NCT0000001’ refers to the study with the NCT_ID: ‘NCT0000001’ (saved in STUDIES.nct_id).  The CONDITIONS dataset may contain multiple records with an NCT_ID of  ‘NCT0000001’ which means this study was defined in ClinicalTrials.gov as being associated with the conditions listed in CONDITIONS with that NCT_ID.</li>
  +      <li>Information in several datasets are also related to information in other datasets.  In this case, the dataset that belongs to another dataset will include a foreign key that identifies the record to which it belongs. Foreign keys are always named according to a simple rule:  the singular name of the related dataset followed by: ‘_id’.   For example, FACILITY_CONTACTS includes a data element: facility_id which is the foreign key to FACILITY.id</li>
  +      <li> Each record’s foreign_key (ie. facility_id) contains the value of the unique identifier (id) of the record in the other dataset to which it belongs. For example, a facility may have multiple contacts.  To find the contacts for a particular facility, look for the records in FACILITY_CONTACTS where the value in facility_id is same as the value in id for that facility in FACILITIES.  In short, datasets are related to each other with this pattern: dataset1.<dataset2_name>_id = dataset2.id </li>
  +
  +    </ul>
  +      <p>Example:</p>
  +      <p>facilities.id = contacts.facility_id  (this finds  all contacts for a facility)</p>
  +      <p>or inversely</p>
  +      <p>contacts.facility_id = facilities.id   (this finds the facility for a particular contact)</p>
  +
  +      <h2>Information about trial sites (FACILITIES and COUNTRIES) </h2>
  +      <p> Information about organizations (trial sites) where the study is/was conducted is stored in the FACILITIES dataset.  This is a snapshot of the facility information that was included in the study record on the date that information was downloaded from ClinicalTrials.gov.  Naturally, study sites initiated after ClinicalTrials.gov data were downloaded to AACT are not included in AACT.</p>
   		<p>AACT also includes a COUNTRIES dataset, which contains one record per unique country per study. Rows in the COUNTRIES dataset come from two sources:</p>
  -		<ul>
  +    <ul class='pointsToConsider'>
   			<li>Countries where study facilities are located as determined by an NLM algorithm that identifies the set of unique countries associated with a study; these are stored in the COUNTRIES dataset with the flag removed=false. </li>
   			<li>In some cases, ClinicalTrials.gov data submitters remove sites from the study record.  Removed sites do not appear in the FACILITIES dataset. If all of a country’s sites have been removed from a study record, the NLM algorithm creates a ‘Removed Country’ record. In AACT, this record is stored in the COUNTRIES dataset with removed=true. </li>
   
  -			<p>The reasons sites are removed from the FACILITIES dataset are varied but unknown.  A site may have been removed because it was never initiated or because it was entered with incorrect information.  The recommended action for sites that have completed or have terminated enrollment is to change the enrollment status to “Completed” or “Terminated”; however, such sites are sometimes deleted from the study record by the responsible party.  Data analysts may consider using Countries where  removed=TRUE to supplement the information about trial locations that is contained in FACILITIES, particularly for studies that have completed enrollment and have no records in FACILITIES.</p>
  -
  -			<p>Users who are interested in identifying countries where participants are being/were enrolled may use either the FACILITIES or COUNTRIES dataset (where COUNTRIES.removed=false) with equivalent results.</p>
  -
  -
  -		<h2>MeSH terms in BROWSE_CONDITIONS and BROWSE_INTERVENTIONS</h2>
  +      <p>The reasons sites are removed from the FACILITIES dataset are varied but unknown.  A site may have been removed because it was never initiated or because it was entered with incorrect information.  The recommended action for sites that have completed or have terminated enrollment is to change the enrollment status to “Completed” or “Terminated”; however, such sites are sometimes deleted from the study record by the responsible party.  Data analysts may consider using Countries where  removed=TRUE to supplement the information about trial locations that is contained in FACILITIES, particularly for studies that have completed enrollment and have no records in FACILITIES.</p>
  +
  +      <p>Users who are interested in identifying countries where participants are being/were enrolled may use either the FACILITIES or COUNTRIES dataset (where COUNTRIES.removed=false) with equivalent results.</p>
  +
  +
  +    <h2>MeSH terms in BROWSE_CONDITIONS and BROWSE_INTERVENTIONS</h2>
       <p>When data submitters provide information to ClinicalTrials.gov about a study, they’re encouraged to use Medical Subject Heading (MeSH) terminology for interventions, conditions, and keywords.  The BROWSE_CONDITIONS and BROWSE_INTERVENTIONS datasets contain MeSH terms generated by an algorithm run by NLM. The NLM algorithm is re-run nightly on all studies in the ClinicalTrials.gov database, and sources the most up-to-date information in the study record, the latest version of the algorithm, and the version of the MeSH thesaurus in use at that time.</p>
   
  -		<h2>>“Delayed Results” data elements available in AACT</h2>
  +    <h2>“Delayed Results” data elements available in AACT</h2>
   
       <p>A responsible party may delay the deadline for submitting results information to ClinicalTrials.gov if one of the following two certification conditions applies to the trial:</p>
   
  -    <ul>
  +    <ul class='pointsToConsider'>
         <li>Initial approval: trial completed before a drug, biologic or device studied in the trial is initially approved, licensed or cleared by the FDA for any use.</li>
         <li>New use: the manufacturer of a drug, biologic or device is the sponsor of the trial and has filed or will file within one year, an application seeking FDA approval, licensure, or clearance of the new use studied in the trial. A responsible party may also request, for good cause, an extension of the deadline for the submission of results.</li>
       </ul>
  
positive.code: 
      <h2>Use of appropriate statistical inference</h2>
  
      <p>If the AACT results data are to be used to support a meta-analysis or systematic review of the safety or efficacy of a particular intervention, then standard methods of meta-analysis or systematic review (e.g., the PRISMA statementxxxxxx[7]) should be used to appropriately account for study-to-study variability and other sources of uncertainty or bias. We recommend that authors consider the following points when deciding whether to report p-values, confidence intervals, or other probability-based inference when performing aggregate analysis of the ClinicalTrials.gov database:  </p>
  
      <h3>Is the data-generating mechanism random?</h3>
      <p>Methods of statistical inference such as p-values and 95% confidence intervals are most appropriate when used to quantify the uncertainty of estimates or comparisons due to a random process that generates the data. Examples of such processes include selection of a random sample of subjects from a broader population, randomly assigning a treatment to a cohort of subjects, or a coin toss about which we aim to predict future results.</p>
  
negative.code: 
    <div class="basicHero">
      <h1>Guide to Using Aggregate Analysis of ClinicalTrials.gov (AACT) for Statistical and Aggregate Analysis</h1>
    </div>
  
  </div>
  
  <section class="pointsToConsider">
    <div class="wrap">
  
      <h2>What is AACT?</h2>
  
      <p>AACT is the database for Aggregate Analysis of <a target='_blank' href='clinicaltrials.gov'>ClinicalTrials.gov</a>. Version 2 of AACT is a postgreSQL relational database containing information about clinical studies that have been been registered at ClinicalTrials.gov. AACT includes all of the protocol and results data elements for studies that are publicly available at ClinicalTrials.gov. Content is downloaded daily from ClinicalTrials.gov and loaded into AACT.</p>
  
      <p> <a target='_blank' href="http://ec2-52-87-233-209.compute-1.amazonaws.com/aact_points_to_consider.docx" download>Download a word document</a> containing the   information on this page.</p>
  
      <h2>What population of studies is represented in AACT?</h2>
  
      <p>All studies registered and publicly available at ClinicalTrials.gov are included in AACT. The ClinicalTrials.gov was released for the registration of studies on February 29, 2000. The registry accepts interventional studies in which participants are assigned according to a research protocol to receive specific interventions, as well as observational studies. It also includes Expanded Access records which describe the procedure for obtaining an experimental drug or device for patients who are not adequately treated by existing therapy and who are unable to participate in a controlled clinical study.</p>
  
      <p>The registration of studies and reporting of results and adverse events has been mandated to a large extent by requirements (both legal and institutional) implemented as part of the Food and Drug Administration Amendments Act (FDAAA), as well as by requirements introduced by the International Committee of Medical Journal Editors (ICMJE) and the European Medicines Agency (EMA) regarding registration of clinical studies. Table 1 describes the scope of these requirements.</p>
  
      <h4>Table 1: Scope of Interventional Studies Covered by Major Reporting Policies*</h4>
  
    <table class='pointsToConsider'>
      <tr>
        <th>Policy Requirements</th>
        <th>Registration & Results Reporting Requirements</th>
        <th>Effective Date</th>
      </tr>
  
      <tr>
        <td><p><a target='_blank' href='http://grants.nih.gov/grants/guide/notice-files/NOT-CA-15-011.html'>NCI Access Policy</a></p></td>
        <td><p>The NCI, NIH, issued its Policy Ensuring Public Availability of Results from NCI-supported Clinical Trials. Generally, for "all initiated or commenced NCI-Supported Interventional Clinical Trials whether extramural or intramural" (Covered Trials), "Final Trial Results are expected to be reported in a publicly accessible manner within 12 months of the Trial's Primary Completion Date regardless of whether the clinical trial was completed as planned or terminated earlier." This policy "will be incorporated as a Term and Condition of the award."</p></td>
        <td><p><b>January, 2015</b></p></td>
    </tr>
  
    <tr>
      <td><a target='_blank' href='http://www.fda.gov/RegulatoryInformation/Legislation/SignificantAmendmentstotheFDCAct/FoodandDrugAdministrationAmendmentsActof2007/default.htm'>FDAAA</a></td>
      <td>The following must be registered in ClinicalTrials.gov:
        <ul class='pointsToConsider'>
          <li>Interventional studies of drugs, biologics, or devices (whether or not approved for marketing)</li>
          <li>Studies phases 2 through 4</li>
          <li>Studies with at least 1 US site or conducted under IND/IDE</li>
        </ul>
  
        <p>Results and adverse event reporting is required for studies that meet the above registration requirements if they study drugs, biologics, or devices that are approved, licensed, or cleared by the FDA.</p>
      </td>
  
      <td>
        <b>September 27, 2007.</b> Studies initiated after this date, or with a completion date later than December 25, 2007 are subject to FDAAA requirements. Registration is required no later than 21 days after first patient is enrolled. Results and adverse events must be reported for these studies (if required) within 1 year of completing data collection for the pre-specified primary outcome.
        <br><br>
        <b>September, 2008.</b> Results reporting launched with optional adverse event reporting.
        <br><br>
        <b>September, 2009.</b>  Adverse event information became required.
      </td>
    </tr>
  
    <tr>
      <td><a target='_blank' href='http://www.icmje.org/about-icmje/faqs/clinical-trials-registration/'>ICMJE</a></td>
      <td>
        Interventional studies of any intervention type, phase, or geographical location must be registered in ClinicalTrials.gov or other approved registry.
        <br>No results reporting requirements.
      </td>
      <td>
        <b>July 1, 2005</b>.  Studies initiated after this date must be registered before first patient enrolled; studies initiated before this date must be retrospectively registered to be considered for publication.
      </td>
    </tr>
  
    <tr>
      <td><a target='_blank' href='http://www.ema.europa.eu/ema/index.jsp?curl=pages/regulation/general/general_content_000629.jsp&mid=WC0b01ac05808768df'>EMA</a></td>
      <td><p>The following must be registered in ClinicalTrials.gov or other approved registry:
        <ul class='pointsToConsider'>
          <li>Interventional studies of drugs or biologics (whether or not approved for marketing)</li>
          <li>Pediatric phase 1 studies;</li>
          <li>Studies in phases 2 through 4</li>
          <li>Studies taking place in at least 1 EU site</li>
        </ul>
        <p>Results reporting required for all studies that meet registration requirements.</p>
      </td>
      <td>
          <p>
            <b>May 1, 2004</b>. EMA launched <a target='_blank' href='https://eudract.ema.europa.eu/'>EudraCT</a>
            <br><br>
            <b>March 22, 2011</b>. The <a target='_blank' href='https://www.clinicaltrialsregister.eu/'>EU Clinical Trials Register</a> was launched by the EMA
         </p>
      </td>
    </tr>
  
  </table>
  
  <p>* Adapted from <a target='_blank' href="http://www.nejm.org/doi/full/10.1056/NEJMsa1012065#t=article">The ClinicalTrials.gov results database – update and key issues</a>. For complete descriptions of policy requirements, see the references cited. EMA denotes European Medicines Agency; EU, European Union; FDAAA, Food and Drug Administration Amendments Act; ICMJE, International Committee of Medical Journal Editors; IDE, investigational device exemption; IND, investigational new drug application, NCI, National Cancer Institute; NIH, National Institutes of Health; US, United States.</p>
  
      <p>Based on these requirements, the following are examples of characteristics that may influence the likelihood that a study is included in the ClinicalTrials.gov registry:</p>
  
      <ul class='pointsToConsider'>
        <li>Interventional studies are more likely to be registered than observational studies.</li>
        <li>Studies that began before the ICMJE requirement in <b>July, 2005</b> are less likely to be registered, especially if their results are unpublished (e.g., negative studies).</li>
        <li>Studies with drug, biological, or device interventions are more likely to be registered than studies of other interventions.</li>
        <li>Studies with at least one site in the United States or European Union are more likely to be registered than studies with no such sites.</li>
        <li>Studies involving a drug or device that is manufactured in the United States are more likely to be registered than studies involving a drug or device manufactured outside of the United States.</li>
  