Psychology & Neuroscience Stack Exchange is a question and answer site for practitioners, researchers, and students in cognitive science, psychology, neuroscience, and psychiatry. It only takes a minute to sign up.
Sign up to join this community
What are the commonly observed biases in survey and are there any statistical or non-statistical methods to avoid it?
I am conducting a survey for automobile seat comfort, I think my current methodology may have following issues :
Subjects may give average rating when they are unsure
Subjective rating may be anchored on the first vehicle subject sits in
There are lots of commonly observed biases in surveys, and a whole field of survey methodology and statistics which investigates them. To get an idea of this, take a look at the table of contents of these journals: http://poq.oxfordjournals.org/content/current http://jssam.oxfordjournals.org/content/current
Many people in this field are dealing with large data-sets (e.g., national censuses with many thousands of respondents), and they concentrate on things like estimating non-responses statistically (to take into account that some people with some opinions are less likely to complete the survey in the first place). Other widely investigated biases include "social desirability", which is the issue that people often respond with what is most social acceptable, even if it is not necessarily accurate.
However, your question suggests you are looking on a smaller scale. Although it is a bit old, this is a nice review of cross-overs between psychology and survey statistics. Not sure if the full article is stuck behind a paywall. This covers some similar ground.
RE: Your first identified issue, I assume that by "average" rating, you mean that people tend to choose the middle scale point if they are unsure (obviously they don't normally know the true average/mean response from their peers). There is evidence of this (see the Schwarz article above). From the abstract of Schwarz et al., 1985:
These results indicate that subjects inferred the average amount of television watching from the response alternatives provided them and used it as a standard of comparison in evaluating their behavior and its implications.
What this means is that you should be a bit careful about choosing your scale (people will use the end points to evaluate what the average rating is). However, normally you are interested in relative deviations from this (which seats are rated higher or lower than average relative to other seats), so this is an issue only if you are seeking a "real number" answer (such as the number of hours people watch TV). One way to counteract the "centering" bias is to use a 4 or 6-point scale rather than an odd number (so that there is no exact centre option) A related issue is that people are often reluctant to choose the most extreme options ("endpoint bias"). The wiki on Likert scales has some information about this, assuming you are using such a scale.
RE: Your second issue, yes, question order matters. People often spend less time and respond differently on later questions (e.g., this paper). The specific issues of anchoring responses are discussed here. However, In your case this should be easy to control for by randomising the order in which they sit in the vehicles.
