How to use Mechanical Turk for longer studies (i.e., 30 minutes+)? I'm interested in using Mechanical Turk as a means of recruiting participants for online psychology studies. I have never used it for any research. However, many of my studies take around 30 minutes to an hour to complete. A typical study might involving answering a set of questionnaires using Inquisit. My impression is that Mechanical Turk works best with short studies (i.e., one to five minutes). 


Is it possible to use Mechanical Turk for longer studies?
Are there any important tips for effectively using Mechanical Turk for longer studies?
Are there any tutorials or case studies which document successful methods for using Mechanical Turk for longer studies?
Or alternatively are there better systems for recruiting paid participants for longer online studies?


In general I appreciate that longer studies would require much more remuneration probably more than just a multiple of the amount of time.
I also imagine that there would be additional issues of quality control.
I've just started reading up on Mechanical Turk. This is a summary of some of the tips that I've found. Admittedly, most of it applies generally to psychological experiments, and not specifically to longer ones.

David Sharek discusses his workflow which explicitly includes studies in the 30 minute range. Thus, this post is one of the most relevant for dealing with the issue of longer studies.

Assorted resources

Here are some  other assorted resources; see also the references at the bottom.


Interoperating Inquisit with other tools
TurkGate is an open source tool to facilitate the use of Mechanical Turk with external psychology experiments.
Mason and Suri (2012) review the potential benefits, implementation, and ethics of Mechanical Turk. 
Michael Buhrmester has a guide for social scientists. He provides an outline of his workflow when running studies. 


Assorted blogs relevant to Mechanical Turk mentioned by Buhrmester


http://behind-the-enemy-lines.blogspot.com. 
http://experimentalturk.wordpress.com.
http://groups.csail.mit.edu/uid/deneme.
http://blog.crowdflower.com.


Configuring external surveys

The general model seems to be to have a link to an external site (make sure it opens in a new tab or window) where the survey is delivered and a box for the completion code to be entered. 

Buhrmester discusses various completion code systems and  opted for the relatively low tech option of getting participants to make up a 4 or 5 digit number and enter it both into the survey and into the mturk. He then uses time stamp data to verify the original completer.

Payment

Mason and Suri state


  However, recent research on the behavior of workers (Chilton et al.,
  2010) demonstrated that workers had a reservation wage (the least
  amount of pay for which they would do the task) of only \$1.38 per
  hour, with an average effective hourly wage of $4.80 for workers
  (Ipeirotis, 2010a).


In terms of the relationship between payment and quality of worker they cite studies suggesting that there is an initial positive relationship that levels off at a certain point such that at a certain point additional payment does not improve performance.
Masson and Suri then suggest:


  Consequently, it is often advisable to start by paying less than the
  expected reservation wage, and then increasing the wage if the rate of
  completed work is too low.


Similarly wages up to a point should increase the speed of data collection.

Rejection of hits

Regarding rejecting hits, Michael has simply accepted all hits. This may be simpler than trying to work out which hits are legitimate. This also made sense given that he was often only paying 10 cents per participant for 10 minute experiments. It also has the benefit of not damaging your reputation.

Quality assurance

There are two issues here. Did the participant complete the study at all? And did they complete the study in an appropriate manner (e.g., trying on performance task; reading instructions properly; etc.)?

A general approach is to incorporate additional means to usual for detecting dodgy data.
If it's simple to filter out such participants then they don't corrupt the final dataset.

A few ideas:


item level reaction time measures 
response patterns to negative and positively worded items
repeat items which should yield identical responses
performance measures
Include very simple true-false questions (e.g., 2+2; Who is the president of the United States); Mason and Suri mention that in 500 responses only six got it wrong and three didn't answer.


Buhrmester makes the causal observation that the quality of responses may vary based on the country of responders, so for example limited participation to US participants is one coarse means of filtering for quality.

Managing reputation as a requester

Buhrmester mentions accepting all hits both for simplicity and managing reptuation.

Mason and Suri (2012) discuss how reputation is discussed and monitored on external sites.


  Turkopticon is a site that allows workers to rate requesters along four
  axes: communicativity, generosity, fairness, and promptness. Turker
  Nation is an online bulletin board where workers routinely comment on
  requesters and communicate about individual HITs. It is strongly
  encouraged that new requesters “introduce” them- selves to the
  Mechanical Turk community by first posting to Turker Nation before
  putting up HITs.


References


Rand, D. G. (2012). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of theoretical biology, 299, 172-179.
Buhrmester, M., Kwang, T., &amp; Gosling, S.D. (2011). Amazon’s Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? Perspectives on Psychological Science, 6(1), 3-5.
Mason, W., &amp; Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior research methods, 44(1), 1-23.
Berinsky, A. J., Huber, G. A., &amp; Lenz, G. S. (2011). Using Mechanical Turk as a subject recruitment tool for experimental research. Submitted for review.
Berinsky, A. J., Huber, G. A., &amp; Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon. com's mechanical turk. Political Analysis, 20(3), 351-368.