Ask HN: Small A/B tests impractical?
I was playing around with numbers to see how many experiments I need to get a statistically significant A/B test result. I used to think that I can solve any small issue in my app by just A/B testing which way is better (patio11 influence), but now it seems that I can't do any tests in a realistic timeframe unless they have more than 2.5 percentage points of effect.<p>If you have a landing page with 10% conversion rate and want to see if your change improved it to 11%, it seems that you need 8000 people in group A and 8000 people in group B. I'm getting about 400 new people daily to try it with, so it becomes impractical to do small tests like these. If I can improve conversion from 10% to 12.5%, then I only need 2600 people, which would mean I could run one test per week.<p>Am I kidding myself if I run tests that aren't statistically significant? Of two options, even if the difference isn't significant, it would still seem the odds are in your favor to pick the one that performed better.<p>I didn't do the math, just used a random calculator I found: http://www.prconline.com/education/tools/statsignificance/index.asp