/ test gotchas an important point want to make is that the results of an / testeven when you measure them in principled manner using -valuesis not gospel there are many effects that can actually skew the results of your experiment and cause you to make the wrong decision let' go through few of these and let you know how to watch out for them let' talk about some gotchas with / tests it sounds really official to say there' -value of percentmeaning there' only percent chance that given experiment was due to spurious results or random variationbut it' still not the be-all and end-all of measuring success for an experiment there are many things that can skew or conflate your results that you need to be aware of soeven if you see pvalue that looks very encouragingyour experiment could still be lying to youand you need to understand the things that can make that happen so you don' make the wrong decisions remembercorrelation does not imply causation even with well-designed experimentall you can say is there is some probability that this effect was caused by this change you made at the end of the daythere' always going to be chance that there was no real effector you might even be measuring the wrong effect it could still be random chancethere could be something else going onit' your duty to make sure the business owners understand that these experimental results need to be interpretedthey need to be one piece of their decision they can' be the be-all and end-all that they base their decision on because there is room for error in the results and there are things that can skew those results and if there' some larger business objective to this changebeyond just driving short-term revenuethat needs to be taken into account as well