title: A/B Test statistics
body: <p>I am trying to do some statistical analysis of different A/B tests to see which alternative is better and have found conflicting information about this.</p>
  
  <p>First, I am interested in a couple different things:</p>
  
  <ul>
  <li>Tests that measure success by counting events, such as conversions or emails sent</li>
  <li>Tests that measure success by counting revenue</li>
  <li>Tests that have only two alternatives (control and new)</li>
  <li>Tests that have multiple alternatives (control and multiple new)</li>
  </ul>
  
  <p>I was hoping to find a simple set of formulae or rules for doing this analysis but have found more questions than answers.</p>
  
  <p><a href="http://blog.asmartbear.com/easy-statistics-for-adwords-ab-testing-and-hamsters.html" rel="nofollow noreferrer">This site</a> says that you can't compare multi-alternative tests; you can only do pairwise comparisons and do a chi-squared analysis to see if the whole test is statistically significant or not.</p>
  
  <p><a href="http://elem.com/~btilly/effective-ab-testing/" rel="nofollow noreferrer">This site</a> Suggests a way to do A/B/C/D testing (starts on slide 74), analysing the results using the G-Test (which it says is related to chi-squared) but isn't clear on the details of using a fudge factor.  It also suggests that you can only use the A/B/C/D approach to eliminate alternatives until you end up with a clear winner in an A/B comparison.</p>
  
  <p><a href="http://20bits.com/articles/statistical-analysis-and-ab-testing/" rel="nofollow noreferrer">This site</a> gives an example of an A/B/C/D test (including control) and shows how to compare the conversion rate to determine a winner.  Unlike <a href="http://elem.com/~btilly/effective-ab-testing/" rel="nofollow noreferrer">this approach</a> it does not recommend eliminating alternatives but rather picks a winner right off the bat (Assuming statistically significant results).</p>
  
  <p>Perhaps I'm naive but I would think that by now a stats analysis library would exist to deal with this very problem.  I would also appreciate more information about what algorithms/equations are needed to solve these problems.  It's been a long time since my university Stats class.</p>