it might even be involved with the weatherduring the summer people behave differently because it' hot out they're feeling kind of lazythey're on vacation more often maybe if you happen to do your experiment during the time of terrible storm in highly populated area that could skew your results as well againjust be cognizant of potential seasonal effectsholidays are big one to be aware ofand always take your experience with grain of salt if they're run during period of time that' known to have seasonality you can determine this quantitatively by actually looking at the metric you're trying to measure as success metricbe itwhatever you're calling your conversion metricand look at its behavior over the same time period last year are there seasonal fluctuations that you see every yearand if soyou want to try to avoid running your experiment during one of those peaks or valleys selection bias another potential issue that can skew your results is selection bias it' very important that customers are randomly assigned to either your control or your treatment groupsyour or group howeverthere are subtle ways in which that random assignment might not be random after all for examplelet' say that you're hashing your customer ids to place them into one bucket or the other maybe there' some subtle bias between how that hash function affects people with lower customer ids versus higher customer ids this might have the effect of putting all of your longtimemore loyal customers into the control groupand your newer customers who don' know you that well into your treatment group what you end up measuring then is just difference in behavior between old customers and new customers as result it' very important to audit your systems to make sure there is no selection bias in the actual assignment of people to the control or treatment group you also need to make sure that assignment is sticky if you're measuring the effect of change over an entire sessionyou want to measure if they saw change on page butover on page they actually did conversionyou have to make sure they're not switching groups in between those clicks soyou need to make sure that within given sessionpeople remain in the same groupand how to define session can become kind of nebulous as well