title: T-Test on one DataFrame with Groups from another DataFrame
body: <p><strong>The Goal:</strong></p>
  
  <p>Perform t-test on DataFrame (df_rna) using group found in another DataFrame (df_cnv).  Reduce the test DataFrame (df_rna) the row indices with the most significant scores from the t-test.</p>
  
  <p><strong>Code Sample:</strong></p>
  
  <pre><code># Dataframe (df_cnv) that forms groups of columns (cells) either\ belonging to True or False for t-test.
  cnv = {'gene': ['x','y','z','n'],
          'cell_a': [0,-1,0,-1],
          'cell_b': [0,-1,-1,-1],
          'cell_c': [-1,0,-1,0],
          'cell_d': [-1,0,-1,0],
          'cell_e': [-1,0,0,0]
         }
  df_cnv = pd.DataFrame(cnv)
  df_cnv.set_index('gene', inplace=True)
  cnv_mask = df_cnv &lt; 0
  cnv_mask  # True values are negative (gene loss is True)
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/9SWcg.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/9SWcg.png" alt="cnv_mask masked for negative values"></a></p>
  
  <pre><code># DataFrame for t-test and subsequent reduction to most significant rows
   rna = {'gene': ['x','y','z','n'],
              'cell_a': [1, 5, 8,9],
              'cell_b': [8, 5, 4,9],
              'cell_c': [8, 6, 1,1],
              'cell_d': [1, 2, 7,1],
              'cell_e': [5, 7, 9,1],
             }
      df_rna = pd.DataFrame(rna)
      df_rna.set_index('gene')
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/evNS1.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/evNS1.png" alt="df_rna"></a></p>
  
  <pre><code># Manually computed T-Tests, save results in DataFrame df_report
  x = scipy.stats.ttest_ind([8,1,5],[1,8])
  y = scipy.stats.ttest_ind([5,5], [6,2,7])
  z = scipy.stats.ttest_ind([4,1,7], [8,9])
  n = scipy.stats.ttest_ind([9,9], [1,1,1])
  
  tStat = [gene.statistic for gene in [x,y,z,n]]
  pVal = [gene.pvalue for gene in [x,y,z,n]]
  
  report = {'gene':['x','y','z','n'],
           't_stat':tStat,
           'p_val':pVal}
  df_report = pd.DataFrame(report)
  df_report.set_index('gene', inplace=True)
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/Qamhg.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Qamhg.png" alt="df_report from t-test"></a></p>
  
  <pre><code># Create reduced version of test DataFrame (df_rna) to contain only rows (genes
  df_pass = df_report.loc[df_report['p_val'] &lt; 0.05]
  passed_genes = set(df_pass.index)
  passed_genes
  
  df_rna_pass = df_rna.loc[df_rna['gene'].isin(passed_genes)]
  df_rna_pass.set_index('gene')
  </code></pre>
  
  <p><a href="https://i.stack.imgur.com/PM8WZ.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/PM8WZ.png" alt="df_rna_pass reduced df_rna DataFrame with only rows with p_val &lt; 0.05"></a></p>
  
  <p><strong>The Question:</strong></p>
  
  <p>Manually setting the t-test groups is not feasible for my large dataset.  How do I compute all the t-test statistics across the whole DataFrame <code>df_rna</code> when the groups of cells being either True and False changes for every row?</p>
  
  <p><strong>Mystery Hang:</strong> (happens if your don't cache the results of <code>rnadf_all[~cnv_mask]</code>)</p>
  
  <pre><code>C:\Users\test\Anaconda3\lib\site-packages\numpy\core\_methods.py:82: RuntimeWarning: Degrees of freedom &lt;= 0 for slice
    warnings.warn("Degrees of freedom &lt;= 0 for slice", RuntimeWarning)
  ---------------------------------------------------------------------------
  KeyboardInterrupt                         Traceback (most recent call last)
  &lt;ipython-input-16-ccabe33b2612&gt; in &lt;module&gt;()
       34 
       35 for r in rnadf_all[cnv_mask].iterrows():
  ---&gt; 36     df_report.at[r[0], 't_stat'], df_report.at[r[0], 'p_val'] = scipy.stats.ttest_ind(r[1].dropna(), rnadf_all[~cnv_mask].loc[r[0]].dropna())
       37 
       38 df_pass = df_report.loc[df_report['p_val'] &lt; 0.05]
  
  C:\Users\test\Anaconda3\lib\site-packages\pandas\core\frame.py in __getitem__(self, key)
     1963             return self._getitem_array(key)
     1964         elif isinstance(key, DataFrame):
  -&gt; 1965             return self._getitem_frame(key)
     1966         elif is_mi_columns:
     1967             return self._getitem_multilevel(key)
  
  C:\Users\test\Anaconda3\lib\site-packages\pandas\core\frame.py in _getitem_frame(self, key)
     2036         if key.values.dtype != np.bool_:
     2037             raise ValueError('Must pass DataFrame with boolean values only')
  -&gt; 2038         return self.where(key)
     2039 
     2040     def query(self, expr, **kwargs):
  
  C:\Users\test\Anaconda3\lib\site-packages\pandas\core\generic.py in where(self, cond, other, inplace, axis, level, try_cast, raise_on_error)
     3931         # try to align
     3932         try_quick = True
  -&gt; 3933         if hasattr(other, 'align'):
     3934 
     3935             # align with me
  
  KeyboardInterrupt: 
  </code></pre>