If you come to this question from the bayesian tradition, then there is only one place where you can sneak in bias: your prior. This dovetails nicely with the wikipedia definition:
a pattern of deviation in judgment that occurs in particular situations, leading to perceptual distortion, inaccurate judgment, illogical interpretation, or what is broadly called irrationality.
Since bayesian updating is considered to be 'rational', the only place to sneak in 'irrationality' (in quotes because we are using these terms very loosely) is in the prior before you were given any evidence/observations. So for a bayesian, measuring bias is measuring the prior.
Measuring a prior
Conveniently, Kalish et al. (2007) have a nice mechanism for measuring people's priors: have $n$ participants: $1, ... , n$ and give the first one some real input-output pairs on the relevant task to learn from. To train the $i + 1$th participant:
- take the $i$th participant,
- give them some inputs and ask them what they think the output should be,
- use the input-output pairs they generate to train the $i+1$th participant.
Then, towards the end of the chain, the participants will start to converge towards their prior or inherent bias.
A real example is of people's bias in functional relationships. The first person is given 25 $(x,y)$ pairs from some function $y = f(x)$. A person at stage $i$ is given 25 $(x,y)$ pairs from the person at stage $i - 1$. The person is then tested by being given an $x$ value and asked for a $y$, 25 times. The results of this are passed on to the person at trial $i + 1$ as the training data. The results:
Show that people have a strong bias towards positive linear relationships. One could apply a similar procedure to other domains, but of course finding a good way to do so is a particular domain is an important research question for an experimental psychologist.
- Kalish, M., Griffiths, T. & Lewandowsky, S. (2007). Iterated learning: Intergenerational knowledge transmission reveals inductive biases. Psychonomic Bulletin & Review, 14, 288-294. doi: 10.3758/BF03194066