Can I use Mechanical Turk for a longitudinal study?

Question

My thesis looks at depression and social support. My design involves participants answering taking a survey every other day for a total of 10 times. Is is the same set of questions that they will answer each time, but I need to see how the responses change over time.

1) Is it possible to set up such a 10 part study on Mechanical Turk, where they will have to return every other day to take the next part?

2) If yes, will have to pay them as they finish each part, or can I pay them when they finish the whole study? I was also wondering if it might be possible to pay a bonus amount to those who finish the whole study - as an incentive to keep coming back each day.

2a) How much should I pay them each time? The survey consists of 23 rating questions on a 5 point Likert scale. I tried it and it took me a little less than 4 minutes.

3) Also, is it possible to accept their answers but only pay the ones who finish the whole study? Because I will be discarding the participants with missing data from my study anyway.

4) The survey results will be anonymous but it is a diary study of sorts. So, I need to be able to connect participants' responses from one day to the next. Will Mechanical Turk allow me to generate a specific code for each participant? Or does each worker already have a code?

5) What survey platform do you suggest I use for setting up my study? Will google docs work?

6) I read somewhere that Mturk workers don't like giving out their emails. But I need to be able to send them emails everytime the responses are due. Will Mechanical Turk allow for this?

Answer 1

The best way to set up a Mechanical Turk longitudinal study is with TurkPrime.com. You can follow-up with users and invite them to take a new survey. As long as they are properly incentivized they will come.

I recommend that you inform them at the start that you are running a 10 part study and that those who perform well on each of the 4 minute studies will also receive a significant bonus, say $5-$10.

See here for some more instructions on setting this up.

Answer 2

Prolific Academic

Alternatively, you could use Prolific Academic. It is similar to Mechanical Turk in that it provides a system for recruiting paid online participants. However, it is more specifically designed for running academic research. And it has specific provision for running longitudinal studies. I've used it for cross-sectional research, and had a fairly good experience.

Here is a post about running longitudinal studies on Prolific.

Payment: Payment on Prolific is based on the proposed time it takes to complete the study and a minimum effective hourly rate. In general, I found that most participants are quite quick at completing survey items. If a participant takes a little longer, they still get the proposed rate. So at present, you might pay say 5/ 60 minutes * 5 pounds = .41 pounds per participant (i.e., about 60 US cents per wave per participant). But this is just a rough guide.

Identification across waves: Prolific assigns each participant a unique ID that would be common across logins. You'd generally just have a question that asks participants to enter their prolific ID at the start of each wave.

Not paying for incomplete data: I'm not sure what is considered reasonable here. I think each wave of data collection would be seen as a separate study, and you would generally be expected to pay if the participant completes the wave. One option would be to increase the payment for subsequent waves or offer bonus payment to incentivise completion.

Survey Platform: Our school has a licence to Qualtrics. I've found this to be very good for surveys and it integrates well with Prolific (i.e., [redirecting participants back to Prolific](https://prolificac.zendesk.com/hc/en-us/articles/202985142-How-do-I-add-a-completion-URL-at-the-end-of-my-study-, copying surveys to create subsequent waves), working many devices, etc.). That said, your study is pretty simple in terms of data collection, so you could probably use just about any survey platform.

Sending emails: You're not allowed to collect identifying information such as emails. I imagine you would need to setup a subsequent study within Prolific for subsequent waves of data collection. Participants are generally notified of the study by email, but you'd need to check how reliable this is.