Response bias in physical and sexual attraction/attractiveness surveys

Question

There are a few potential sources of response bias in surveys intended to measure attractiveness levels of individuals, especially in the context of differences in personal qualities of judges.

Subjective interpretation of the term

Perhaps, you have heard something along the lines "He is attractive, but he is not my type". In this case the term "Attractiveness" is understood in the context of socially conventional judgement: "Many people would find him attractive".

In other cases, when people are asked about attractiveness of the person, they might assume that they are asked to rate the physical attractiveness of the person from their personal lens.

In order to remove this confusion, the questions can be more direct, such as "Rate the objective level of attractiveness" or "How attracted are you to this person?" Is there any data on how much such a different formulation of the question changes the responses?

Subjective interpretation of the scale

Some surveys use 5-point Likert-type scales, some use 7-point scales. On top of that, the items on the scale can be named differently: they can be raw numbers, or they can be named "Highly [un]attractive", "Neutral" or anyhow differently.

In a numerical scale a person might assume that the lowest score item, 1, stands for "Not attractive" and everything above that signifies higher and higher levels of attractiveness. While such an misinterpretation would be far less likely when the lowest score item was named "Highly unattractive". Is there any data showing how much the difference in the scale used affects the response?

Unemotional response bias

Not exactly certain about the name of the phenomenon, however I will describe it in the following way: A bias produced when the person abstracts too much from the task and forgets about their emotions. For example, consider the question:

Describe your reaction towards the idea of having sex with this person.

With the following options for a response:

1. Strong disgust.
2. Moderate disgust.
3. Mild disgust.
4. Neutral.
5. Mild excitation.
6. Moderate excitation.
7. Strong excitation.

The responses to this question still might be different from the questions asking to rate sexual attraction or attractiveness directly. However, is there any data examining this bias within the context of rating attraction?

Answer 1

Most actual attractiveness and other types of sensitive "excitation/disgust" (using your language term) surveys are done by private organizations or firms across many years, regions and cultures, thus the findings are mostly proprietary and probably hard to retrieve publicly to reference, especially matching your exact keywords specification for comparison. Maybe you need to do some pilot survey yourself to find out some preliminary result.

However, your main concern out of several questions here could be addressed on a general level if you become familiar with some detailed design and planning guidelines for these kind of observational survey samples to reduce response bias as much as possible. which is an important topic especially in social sciences. You may read Kalton and Schuman's The Effect of the Question on Survey Responses: A Review for an introduction.

According to Kalton & Schuman's review, we have to first distinguish factual vs non-factual (opinion) questions in your survey to be aware of different types of possible response bias albeit they may overlap in some cases. Thus your concerned "attractiveness" question is arguably intrinsically non-factual opinion-based in which case according to the authors marginal wording format changes may cause markedly discrepant marginal distributions. A survey that asks “Are you in favor of building a new town hall that will cost taxpayers $2 billion?” is likely to get a very different response from one that is worded as: “Are you in favor of building a new town hall to replace the old one that is contaminated with asbestos?”. However, it seems in your proposed cases ("rate the person's attractiveness" vs "how attracted are you to this person?") there's no such dramatic wording effect possible. It could be related to the issue of open and closed questions mentioned in the non-factual question review section:

The main context in which open questions are used extensively is when the potential responses are both nominal in nature and sizeable in number. These conditions occur often with motivation questions, asking for the principal or all reasons for an occurrence, and with questions asking for the choice of the most, or several most, important factors involved in an issue... The authors present indirect evidence suggesting that when open and properly constructed closed forms of questions yield different responses, the responses to the closed questions are sometimes more valid in their classification of respondents and in describing relationships of the responses with other variables.

Thus ideally you may have to do some pilot open question survey first to distill those appropriate closed options related to your "attractiveness" as there're apparently many different aspects (muscle, body ratio, body organs, etc) affect attractiveness for different people which you may need to reflect in your closed options. Of course I get from your question that you may regard "attractiveness" as some socially accepted "objective" fact and you may have to state this explicitly in your question, then this becomes a factual question and other issues such as social desirability bias with significant cultural variances need to be further addressed. But arguably "attractiveness" of a specific person is always opinion based without factual basis even there may be small variance around its population mean.

For your remaining closed question format concerns about this largely non-factual question could be addressed by Kalton & Schuman's Treatment of Don't Knows and Middle alternatives sections.

Two examples given by Schuman and Presser (1980)illustrate that many respondents will indeed choose one of the alternatives offered for an opinion question even though they do not know about the issue involved... As a way of screening out respondents without opinions, some type of filtering may be used. One possibility is to include an explicit "no opinion" option or filter in the response categories offered to respondents-a "quasi-filter"; in the Schuman and Presser experiment, this offer reduced the proportion of respondents expressing opinions on the two laws to 10 per cent or less. A more forceful possibility is a preliminary filter question "Do you have an opinion on ...?"-a "full filter".

As might be expected, the explicit offer of a middle alternative often substantially increases the proportion of respondents stating a neutral view. In a series of experiments conducted by Kalton et al. (1980), the increases were between 15 and 49 per cent; in a series reported by Presser and Schuman (1980) the increases were between 10 and 20 per cent. Presser and Schuman observe that in their studies and earlier ones involving three point scales (pro, neutral and anti) the increase in support for the neutral view with the offered question form came proportionately from the polar positions, so that the balance between pro's and anti's was not affected by the variation in alternatives offered. This comforting finding failed to hold, however, in two of the three experiments with three-point scales reported by Kalton et al.

So in summary survey result may become very sensitive and subtle to the inclusion of closed options such as "no opinion" and "neutral", and to finally decide your specific wording of those choices expressed in your last two sections (such as employing sensitive "excitation/disgust" or "attraction/repulsion" rhetoric using your term), you'd better initially conduct some pilot study using open question format for your target population if you cannot find reputable data sources to confidently rely on. It's well-known that high quality unbiased surveys are not cheap, nowadays you can perhaps easily and cheaply do some anonymous surveys via internet but many other issues and biases such as the famous non-response bias would occur.

Reference: Kalton, Graham; Schuman, Howard (1982). The Effect of the Question on Survey Responses: A Review. Journal of the Royal Statistical Society. 145 (1): 42–73.