detecting outliers common problem with real-world data is outliers you'll always have some strange usersor some strange agents that are polluting your datathat act abnormally and atypically from the typical user they might be legitimate outliersthey might be caused by real people and not by some sort of malicious trafficor fake data so sometimesit' appropriate to remove themsometimes it isn' make sure you make that decision responsibly solet' dive into some examples of dealing with outliers for exampleif ' doing collaborative filteringand ' trying to make movie recommendations or something like thatyou might have few power users that have watched every movie ever madeand rated every movie ever made they could end up having an inordinate influence on the recommendations for everybody else you don' really want handful of people to have that much power in your system sothat might be an example where it would be legitimate thing to filter out an outlierand identify them by how many ratings they've actually put into the system ormaybe an outlier would be someone who doesn' have enough ratings we might be looking at web log datalike we saw in our example earlier when we were doing data cleaningoutliers could be telling you that there' something very wrong with your data to begin with it could be malicious trafficit could be botsor other agents that should be discarded that don' represent actual human beings that you're trying to model if someone really wanted the mean average income in the united states (and not the median)you shouldn' just throw out donald trump because you don' like him you know the fact ishis billions of dollars are going to push that mean amount upeven if it doesn' budge the median sodon' fudge your numbers by throwing out outliers but throw out outliers if it' not consistent with what you're trying to model in the first place nowhow do we identify outlierswellremember our old friend standard deviationwe covered that very early in this book it' very useful tool for detecting outliers you canin very principled mattercompute the standard deviation of dataset that should have more or less normal distribution if you see data point that' outside of one or two standard deviationsthere you have an outlier rememberwe talked earlier too about the box and whisker diagrams tooand those also have built-in way of detecting and visualizing outliers those diagrams define outliers as lying outside the interquartile range