you need to make sure that your business owners understand that this is an important effect that you need to quantify and understand before making business decisions following an / test or any experiment that you run on the web nowsometimes you need to choose conversion metric that has less variance it could be that the numbers on your website just mean that you would have to run an experiment for years in order to get significant result based on something like revenue or amount spent sometimes if you're looking at more than one metricsuch as order amount or order quantitythat has less variance associated with ityou might see signal on order quantity before you see signal on revenuefor example at the end of the dayit ends up being judgment call if you see significant lift in order quantities and maybe not-so-significant lift in revenuethen you have to say "welli think there might be something real and beneficial going on here howeverthe only thing that statistics and data size can tell youare probabilities that an effect is real it' up to you to decide whether or not it' real at the end of the day solet' talk about how to do this in more detail the key takeaway here isjust looking at the differences in means isn' enough when you're trying to evaluate the results of an experimentyou need to take the variance into account as well -test and -value how do you know if change resulting from an / test is actually real result of what you changedor if it' just random variationwellthere are couple of statistical tools at our disposal called the -test or -statisticand the -value let' learn more about what those are and how they can help you determine whether an experiment is good or not the aim is to figure out if result is real or not was this just result of random variance that' inherent in the data itselfor are we seeing an actualstatistically significant change in behavior between our control group and our test groupt-tests and -values are way to compute that rememberstatistically significant doesn' really have specific meaning at the end of the day it has to be judgment call you have to pick probability value that you're going to accept of result being real or not but there' always going to be chance that it' still result of random variationand you have to make sure your stakeholders understand that