I saw this question on Twitter from Jay Van Bavel (@jayvanbavel), and thought it was a good fit for the site.
Does anyone find that multiple subjects from the same IP address in their #MTurk data? How do you deal with this issue? Citations welcome.
Thus, to rephrase:
Since this is a relatively new problem for behavioral researchers, I don't know that there is a common consensus. I found two articles, one of which was a study that had used crowdsourcing for medical pictograms.
Their approach was as follows:
First, we checked for duplicate records. After sorting the data by participants’ IP addresses, we found three pairs of responses with the same IP address. In two pairs, the pictogram interpretations and the demographic survey answers were nearly identical, but the participation dates were different. We counted them as duplicate records and kept only the first record of each on file.
From:
Yu B, Willis M, Sun P, Wang J (2013) Crowdsourcing Participatory Evaluation of Medical Pictograms Using Amazon Mechanical Turk J Med Internet Res, 15(6):e108 [FREE] [DOI]
In an article just posted this month, the subject is dealt with in more general terms, and some statistics are offered as to how frequently this might occur, and some of the reasons behind it.
Although workers may be able to have more than one concurrent MTurk account and, thus, more than one WorkerID, this is uncommon. Amazon actively works to identify and eliminate duplicate accounts. More important, requesters often restrict lucrative HITs to workers who have completed a large volume of high-quality work in the past
So, it seems that WorkerID can be used as a unique identifier and Amazon does actually screen for duplicate accounts, eliminating some of the risk of duplicate responders to the same study.
In terms of IP addresses, the article offers some idea of the extent of the problem:
Examinations of worker IP addresses typically reveal a small minority of workers (around 2.5 %; Berinsky et al.,2012**) who submit HITs from the same IP address, which may often result from workers being separate members of a single household.
From:
Chandler, J, Mueller, P, Paolacci, G (2013). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, published online 9 July 2013 [DOI]
The first article pointed out the fact that dropping data from a study based upon duplicate IP addresses can be done without much of a cost penalty.
Other demographic factors can be used to discern whether these are different people, but the accuracy of these answers is not guaranteed, but can be bolstered by having dependencies between demographic categories (e.g., recording gender and last menstrual period should be consistent).
Results should be checked more carefully for users that share the same IP address, as even if they are different members of a household, they may be sharing a set of answers, but it seems like there is a low cost to including data from those with matching IP addresses.
** The citation for the Berinsky paper included in the Chandler 2013 work is as follows: Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis, 20(3), 351–368. [DOI] I did not examine this work
I'll address just the first of your three sub-questions, the others have been answered by Chuck Sherrington.
IP addresses are rarely "fixed" to an individual computer. Each provider has a range of IP addresses available and assigns them to computers as they connect to the network. Every time you boot your computer, every time you connect to a WLAN network, you'll probably have a different IP address. Try it with a tool such as http://whatismyipaddress.com
So of course the IP address you had yesterday will be assigned to someone else tomorrow. It only means that both computers were connected through the same provider. No other conclusions can be drawn from the IP address!
(Only the combination of IP address and connection time is unique to your computer. Your provider stores your connection data for a legally specified period, and during this time the police will be able to identify your computer if they know the time of the connection, but this data is not publicly available.)
Usually IP addresses remain stable through a session, but you cannot even rely on that. DSL connections are often reset once every 24 hours, so if you are online at night, you might experience a loss of your network connection, and after that you'll very likely have a different IP. Mobile connections are separated much more often, and your IP address will change as often. So, if you save a user's IP address at the beginning of a survey and again at the end, both might differ! This will be especially true for users surfing through anonymisation tools such as a TOR network which are programmed to switch IP addresses regularly.
You cannot reliably identify a single computer through an IP address. Use cookies for that or rely on user specified data.
See https://en.wikipedia.org/wiki/IP_address#IP_address_assignment