I saw this question on Twitter from Jay Van Bavel (@jayvanbavel), and thought it was a good fit for the site.Does anyone find that multiple subjects from the same IP address in  their #MTurk data? How do you deal with this issue? Citations welcome.Thus, to rephrase:What does it mean when you get two or more Mechanical Turk participants in a study with the same IP address?What is a general strategy for dealing with such cases?Are there are any references discussing the issue?
How should multiple Mechanical Turk research subjects from the same IP address be handled?
methodology;internet;mechanical turk
Since this is a relatively new problem for behavioral researchers, I don't know that there is a common consensus.  I found two articles, one of which was a study that had used crowdsourcing for medical pictograms.  Their approach was as follows:First, we checked for duplicate records. After sorting the data by participants IP addresses, we found three pairs of responses with the same IP address. In two pairs, the pictogram interpretations and the demographic survey answers were nearly identical, but the participation dates were different. We counted them as duplicate records and kept only the first record of each on file.From:Yu B, Willis M, Sun P, Wang J (2013)  Crowdsourcing Participatory Evaluation of Medical Pictograms Using Amazon Mechanical Turk  J Med Internet Res, 15(6):e108   [FREE] [DOI]In an article just posted this month, the subject is dealt with in more general terms, and some statistics are offered as to how frequently this might occur, and some of the reasons behind it.Although workers may be able to have more than one concurrent MTurk account and, thus, more than one WorkerID,  this is uncommon. Amazon actively works to identify and  eliminate duplicate accounts. More important, requesters  often restrict lucrative HITs to workers who have completed  a large volume of high-quality work in the pastSo, it seems that WorkerID can be used as a unique identifier and Amazon does actually screen for duplicate accounts, eliminating some of the risk of duplicate responders to the same study.In terms of IP addresses, the article offers some idea of the extent of the problem:Examinations of worker IP addresses typically reveal a small minority of  workers (around 2.5 %; Berinsky et al.,2012**) who submit HITs from the same IP address, which may often result from workers being separate members of a single household. From:Chandler, J, Mueller, P, Paolacci, G (2013). Nonnavet among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers.  Behavior Research Methods, published online 9 July 2013 [DOI]The first article pointed out the fact that dropping data from a study based upon duplicate IP addresses can be done without much of a cost penalty.  Other demographic factors can be used to discern whether these are different people, but the accuracy of these answers is not guaranteed, but can be bolstered by having dependencies between demographic categories (e.g., recording gender and last menstrual period should be consistent).Results should be checked more carefully for users that share the same IP address, as even if they are different members of a household, they may be sharing a set of answers, but it seems like there is a low cost to including data from those with matching IP addresses. ** The citation for the Berinsky paper included in the Chandler 2013 work is as follows: Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating onlinelabor markets for experimental research: Amazon.comsMechanical Turk. Political Analysis, 20(3), 351368. [DOI]  I did not examine this work 