I've just started reading up on Mechanical Turk. This is a summary of some of the tips that I've found. Admittedly, most of it applies generally to psychological experiments, and not specifically to longer ones.
David Sharek discusses his workflow which explicitly includes studies in the 30 minute range. Thus, this post is one of the most relevant for dealing with the issue of longer studies.
Assorted resources
Here are some other assorted resources; see also the references at the bottom.
Assorted blogs relevant to Mechanical Turk mentioned by Buhrmester
Configuring external surveys
The general model seems to be to have a link to an external site (make sure it opens in a new tab or window) where the survey is delivered and a box for the completion code to be entered.
Buhrmester discusses various completion code systems and opted for the relatively low tech option of getting participants to make up a 4 or 5 digit number and enter it both into the survey and into the mturk. He then uses time stamp data to verify the original completer.
Payment
Mason and Suri state
However, recent research on the behavior of workers (Chilton et al.,
2010) demonstrated that workers had a reservation wage (the least
amount of pay for which they would do the task) of only \$1.38 per
hour, with an average effective hourly wage of $4.80 for workers
(Ipeirotis, 2010a).
In terms of the relationship between payment and quality of worker they cite studies suggesting that there is an initial positive relationship that levels off at a certain point such that at a certain point additional payment does not improve performance.
Masson and Suri then suggest:
Consequently, it is often advisable to start by paying less than the
expected reservation wage, and then increasing the wage if the rate of
completed work is too low.
Similarly wages up to a point should increase the speed of data collection.
Rejection of hits
Regarding rejecting hits, Michael has simply accepted all hits. This may be simpler than trying to work out which hits are legitimate. This also made sense given that he was often only paying 10 cents per participant for 10 minute experiments. It also has the benefit of not damaging your reputation.
Quality assurance
There are two issues here. Did the participant complete the study at all? And did they complete the study in an appropriate manner (e.g., trying on performance task; reading instructions properly; etc.)?
A general approach is to incorporate additional means to usual for detecting dodgy data.
If it's simple to filter out such participants then they don't corrupt the final dataset.
A few ideas:
- item level reaction time measures
- response patterns to negative and positively worded items
- repeat items which should yield identical responses
- performance measures
- Include very simple true-false questions (e.g., 2+2; Who is the president of the United States); Mason and Suri mention that in 500 responses only six got it wrong and three didn't answer.
Buhrmester makes the causal observation that the quality of responses may vary based on the country of responders, so for example limited participation to US participants is one coarse means of filtering for quality.
Managing reputation as a requester
Buhrmester mentions accepting all hits both for simplicity and managing reptuation.
Mason and Suri (2012) discuss how reputation is discussed and monitored on external sites.
Turkopticon is a site that allows workers to rate requesters along four
axes: communicativity, generosity, fairness, and promptness. Turker
Nation is an online bulletin board where workers routinely comment on
requesters and communicate about individual HITs. It is strongly
encouraged that new requesters “introduce” them- selves to the
Mechanical Turk community by first posting to Turker Nation before
putting up HITs.
References
- Rand, D. G. (2012). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of theoretical biology, 299, 172-179.
- Buhrmester, M., Kwang, T., & Gosling, S.D. (2011). Amazon’s Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? Perspectives on Psychological Science, 6(1), 3-5.
- Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior research methods, 44(1), 1-23.
- Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2011). Using Mechanical Turk as a subject recruitment tool for experimental research. Submitted for review.
- Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon. com's mechanical turk. Political Analysis, 20(3), 351-368.
