How can we effectively measure the impact of our data decisions Apologies if this is very broad question, what I would like to know is how effective is A/B testing (or other methods) of effectively measuring the effects of a design decision.

For instance we can analyse user interactions or click results, purchase/ browse decisions and then modify/tailor the results presented to the user.

We could then test the effectiveness of this design change by subjecting 10% of users to the alternative model randomly but then how objective is this?

How do we avoid influencing the user by the model change, for instance we could decided that search queries for 'David Beckham' are probably about football so search results become biased towards this but we could equally say that his lifestyle is just as relevant but this never makes it into the top 10 results that are returned.

I am curious how this is dealt with and how to measure this effectively.

My thoughts are that you could be in danger of pushing a model that you think is correct and the user obliges and this becomes a self-fulfilling prophecy.

I've read an article on this: http://techcrunch.com/2014/06/29/ethics-in-a-data-driven-world/ and also the book: http://shop.oreilly.com/product/0636920028529.do which discussed this so it piqued my interest.
In A/B testing, bias is handled very well by ensuring visitors are randomly assigned to either version A or version B of the site.  This creates independent samples drawn from the same population.  Because the groups are independent and, on average, only differ in the version of the site seen, the test measures the effect of the design decision.

Slight aside: Now you might argue that the A group or B group may differ in some demographic.  That commonly happens by random chance.  To a certain degree this can be taken care of by covariate adjusted randomization.  It can also be taken care of by adding covariates to the model that tests the effect of the design decision.  It should be noted that there is still some discussion about the proper way to do this within the statistics community.  Essentially A/B testing is an application of a Randomized Control Trial to website design.  Some people disagree with adding covariates to the test.  Others, such as Frank Harrel (see Regression Modeling Strategies) argue for the use of covariates in such models.

I would offer the following suggestions:


Design the study in advance so as to take care of as much sources of bias and variation as possible.   
Let the data speak for itself.  As you get more data (like about searches for David Beckham), let it dominate your assumptions about how the data should be (as how the posterior dominates the prior in Bayesian analysis when the sample size becomes large).    
Make sure your data matches the assumptions of the model.