I recently had a phone interview with a consumer tech company for a quant position. The question was basically, imagine a facebook style social network site. Six months ago a new feature called 'mentions' was added which allows you to tag your friends with an @ sign. How would determine whether this feature was a success?I was a bit taken aback by how broad the question was. I first asked if the feature was given to everyone in the network or a sample, to which the interviewer responded you decide - meaning I could approach the analysis either way. I talked in general terms about calculating week over week usage of the feature as well as month over month. I also discussed computing a baseline metric for product interaction and then comparing the usage of the new mentions feature relative the baseline statistic. Overall I left the interview feeling quite dumb, as I have a pretty solid command of stats, but came away looking like an idiot.Are there specific statistical procedures to test for something like this? al la A/B testing, or some kind of hypothesis test? And secondly, is there a good framework for approaching these types of open ended case study style questions in general?
How to approach a Data Science case study question?
statistics;beginner;ab test
This question (something I've asked variants of several times in interviews) has absolutely nothing to do with statistical or other quantitative procedures. What is being asked here is for an understanding of the overall data mining process itself.  The first thing to determine is what the definition of success. So you have to ask. The stakeholder usually will not volunteer this unless asked anyway. Then, depending on the answer describe the overall process for data mining based on this end goal.