title: McNemar's test in Python and comparison of classification machine learning models
body: <p>Is there a good McNemar's test implemented in Python? I don't see it anywhere in Scipy.stats or Scikit-Learn. I may have overlooked some other good packages. Please recommend.</p>
  
  <p>McNemar's Test is almost THE test for comparing two classification algorithms/models given a holdout test set (not through K-fold or resampling method to mimic a test set). Two common alternatives are: t-test for comparing directly true positive proportions <code>p_A</code> and <code>p_B</code> from two algorithms and models <code>A</code> and <code>B</code> by 1) assuming variances follow binomial distributions or 2) estimating variances using repetitive resampling train and test sets. </p>
  
  <p>The latter two, however, have been shown having high Type-1 errors (claims models are statistically different but in essence they are the same). McNemar's test is still considered the best if comparing two classification algorithms or models. See <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=2&amp;cad=rja&amp;uact=8&amp;ved=0ahUKEwii54DM3LPRAhWHgFQKHR5ZAqMQFggkMAE&amp;url=http%3A%2F%2Fwww.cs.iastate.edu%2F~honavar%2Fdietterich98approximate.pdf&amp;usg=AFQjCNFoEqZMm7DktvpNO0T88xZJs8AMgA&amp;bvm=bv.142059868,d.cGw" rel="noreferrer">Dietterich10</a>.</p>
  
  <p>Or as an alternative, how do people statistically compare two classification models in practice if not by McNemar's test?</p>