novelty effects one problem is novelty effects one major achilles heel of an / test is the short time frame over which they tend to be runand this causes couple of problems first of allthere might be longer-term effects to the changeand you're not going to measure thosebut alsothere is certain effect to just something being different on the website for instancemaybe your customers are used to seeing the orange buttons on the website all the timeand if blue button comes up and it catches their attention just because it' different howeveras new customers come in who have never seen your website beforethey don' notice that as being differentand over time even your old customers get used to the new blue button it could very well be that if you were to make this same test year laterthere would be no difference or maybe they' be the other way around could very easily see situation where you test orange button versus blue buttonand in the first two weeks the blue button wins people buy more because they are more attracted to itbecause it' different but year goes byi could probably run another web lab that puts that blue button against an orange button and the orange button would winagainsimply because the orange button is differentand it' new and catches people' attention just for that reason alone for that reasonif you do have change that is somewhat controversialit' good idea to rerun that experiment later on and see if you can actually replicate its results that' really the only way know of to account for novelty effectsactually measure it again when it' no longer novelwhen it' no longer just change that might capture people' attention simply because it' different and thisi really can' understate the importance of understanding this this can really skew lot of resultsit biases you to attributing positive changes to things that don' really deserve it being different in and of itself is not virtueat least not in this context seasonal effects if you're running an experiment over christmaspeople don' tend to behave the same during christmas as they do the rest of the year they definitely spend their money differently during that seasonthey're spending more time with their families at homeand they might be little bitkind of checked out of workso people have different frame of mind