How to use Mechanical Turk for longer studies (i.e., 30 minutes+)?

I'm interested in using Mechanical Turk as a means of recruiting participants for online psychology studies. I have never used it for any research. However, many of my studies take around 30 minutes to an hour to complete. A typical study might involving answering a set of questionnaires using Inquisit. My impression is that Mechanical Turk works best with short studies (i.e., one to five minutes).

• Is it possible to use Mechanical Turk for longer studies?
• Are there any important tips for effectively using Mechanical Turk for longer studies?
• Are there any tutorials or case studies which document successful methods for using Mechanical Turk for longer studies?
• Or alternatively are there better systems for recruiting paid participants for longer online studies?

In general I appreciate that longer studies would require much more remuneration probably more than just a multiple of the amount of time. I also imagine that there would be additional issues of quality control.

• There are no real restrictions on what you can do for an mturk study; all experiment code can be hosted and run on your own servers. – Jeff May 8 '13 at 4:09
• What is the main motivation for using mturk (or generally, online workforce)? Is it because you need more participants than you can recruit using conventional methods? Because you want a long study that won't require your participants to come to the lab every day? Some other reason? – Ofri Raviv May 8 '13 at 7:54
• @ofri I imagine speed of recruitment and cost effectiveness are the two main motivations. For example, If I could get 200 conscientious participants to complete a 30 minute study for US\$5 or US\$10 per participant, and it could all be finalised in a day or so, that would make my life a lot easier. mturk manages payment and it also manages the mapping of free time to tasks. – Jeromy Anglim May 8 '13 at 8:03
• @jeff It's good that it's possible, but I get the impression that there might be additional issues with longer studies around motivating participants to complete the task in the first place and ensuring that they perform the task conscientiously. – Jeromy Anglim May 8 '13 at 8:09

I've just started reading up on Mechanical Turk. This is a summary of some of the tips that I've found. Admittedly, most of it applies generally to psychological experiments, and not specifically to longer ones.

David Sharek discusses his workflow which explicitly includes studies in the 30 minute range. Thus, this post is one of the most relevant for dealing with the issue of longer studies.

Assorted resources

Here are some other assorted resources; see also the references at the bottom.

Assorted blogs relevant to Mechanical Turk mentioned by Buhrmester

Configuring external surveys

The general model seems to be to have a link to an external site (make sure it opens in a new tab or window) where the survey is delivered and a box for the completion code to be entered.

Buhrmester discusses various completion code systems and opted for the relatively low tech option of getting participants to make up a 4 or 5 digit number and enter it both into the survey and into the mturk. He then uses time stamp data to verify the original completer.

Payment

Mason and Suri state

However, recent research on the behavior of workers (Chilton et al., 2010) demonstrated that workers had a reservation wage (the least amount of pay for which they would do the task) of only \$1.38 per hour, with an average effective hourly wage of$4.80 for workers (Ipeirotis, 2010a).

In terms of the relationship between payment and quality of worker they cite studies suggesting that there is an initial positive relationship that levels off at a certain point such that at a certain point additional payment does not improve performance. Masson and Suri then suggest:

Consequently, it is often advisable to start by paying less than the expected reservation wage, and then increasing the wage if the rate of completed work is too low.

Similarly wages up to a point should increase the speed of data collection.

Rejection of hits

Regarding rejecting hits, Michael has simply accepted all hits. This may be simpler than trying to work out which hits are legitimate. This also made sense given that he was often only paying 10 cents per participant for 10 minute experiments. It also has the benefit of not damaging your reputation.

Quality assurance

There are two issues here. Did the participant complete the study at all? And did they complete the study in an appropriate manner (e.g., trying on performance task; reading instructions properly; etc.)?

A general approach is to incorporate additional means to usual for detecting dodgy data. If it's simple to filter out such participants then they don't corrupt the final dataset.

A few ideas:

• item level reaction time measures
• response patterns to negative and positively worded items
• repeat items which should yield identical responses
• performance measures
• Include very simple true-false questions (e.g., 2+2; Who is the president of the United States); Mason and Suri mention that in 500 responses only six got it wrong and three didn't answer.

Buhrmester makes the causal observation that the quality of responses may vary based on the country of responders, so for example limited participation to US participants is one coarse means of filtering for quality.

Managing reputation as a requester

Buhrmester mentions accepting all hits both for simplicity and managing reptuation.

Mason and Suri (2012) discuss how reputation is discussed and monitored on external sites.

Turkopticon is a site that allows workers to rate requesters along four axes: communicativity, generosity, fairness, and promptness. Turker Nation is an online bulletin board where workers routinely comment on requesters and communicate about individual HITs. It is strongly encouraged that new requesters “introduce” them- selves to the Mechanical Turk community by first posting to Turker Nation before putting up HITs.

References

• Rand, D. G. (2012). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of theoretical biology, 299, 172-179.
• Buhrmester, M., Kwang, T., & Gosling, S.D. (2011). Amazon’s Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? Perspectives on Psychological Science, 6(1), 3-5.
• Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior research methods, 44(1), 1-23.
• Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2011). Using Mechanical Turk as a subject recruitment tool for experimental research. Submitted for review.
• Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon. com's mechanical turk. Political Analysis, 20(3), 351-368.

You may want to take a look at SurveyComet.com and TurkPrime.com. Both offer a pretty nice set of tools to run externally hosted surveys (like on Qualtrics and SurveyMonkey) and allow you to create follow up surveys, exclude previous workers, have specific demographic Panels and much more.

A long survey can be broken into two parts. Launch part 1 and later part2 as a follow up which will only be open to those who took part 1.

Disclaimer: I am part of the software development team and am currently working on those sites.

• A quick look suggests the pricing seems reasonable. I'd be curious to know: (a) what proportion of participants tend to complete longer surveys conscientiously - i.e., not skipping questions or responding randomly; (b) whether you can use external tools that require installation of a plugin - in particular, Inquisit is a powerful tool for running online psych experiments, but it does require the user to install a plugin. – Jeromy Anglim Jan 14 '15 at 3:06
• A. SurveyComet workers are monitored to ensure a high degree of reliability and internal tests have shown that random submissions are rare (<1%). B. Mechanical Turk does not allow requesters to ask workers to install programs so using inquisit is a violation of Amazon's terms of service – Jonathan Jan 14 '15 at 21:02
• @JeromyAnglim Although you may possibly get around the MTurk Terms of Service requirement by stating that the HIT is only open to Workers who have previously downloaded Inquisit. So the download is not part of the HIT. – Jonathan Jan 30 '15 at 15:36