nowthese are all issues that using an established off-the-shelf framework like google experiments or optimizely or one of those guys can help with so that you're not reinventing the wheel on all these problems if your company does have homegrowninhouse solution because they're not comfortable with sharing that data with outside companiesthen it' worth auditing whether there is selection bias or not auditing selection bias issues one way for auditing selection bias issues is running what' called an / testlike we saw earlier soif you actually run an experiment where there is no difference between the treatment and controlyou shouldn' see difference in the end result there should not be any sort of change in behavior when you're comparing those two things an / test can be good way of testing your / framework itself and making sure there' no inherent bias or other problemsfor examplesession leakage and whatnotthat you need to address data pollution another big problem is data pollution we talked at length about the importance of cleaning your input dataand it' especially important in the context of an / test what would happen if you have robota malicious crawler that' crawling through your website all the timedoing an unnatural amount of transactionswhat if that robot ends up getting either assigned to the treatment or the controlthat one robot could skew the results of your experiment it' very important to study the input going into your experiment and look for outliersthen analyze what those outliers areand whether they should they be excluded are you actually letting some robots leak into your measurements and are they skewing the results of your experimentthis is veryvery common problemand something you need to be cognizant of there are malicious robots out therethere are people trying to hack into your websitethere are benign scrapers just trying to crawl your website for search engines or whatnot there are all sorts of weird behaviors going on with websiteand you need to filter out those and get at the people who are really your customers and not these automated scripts that can actually be very challenging problem yet another reason to use off-the-shelf frameworks like google analyticsif you can