How to answer questions that don't seem to be possible to be answered on a Strongly disagree - Strongly agree scale and what's their point?

Question

I'm continuously being perplexed by psychological tests that display a lot of questions that don't seem to be able to be answered in a simplistic ways, but nevertheless simplistic answers are demanded.

I was subjected to such tests a few times in my life already. Each time the questions were a major challenge for me... until, out of necessity, I was stopping caring about the accuracy of the answers. Instead, I was giving rough estimations, feeling these estimations are almost meaningless.

As an example, I was once subjected to a test whether I was aggressive, submissive, assertive, or manipulative. The test contained questions like "do I think cunning is an effective way to get what one wants"? Do I think so? Yes, I think it is an effective way insofar people were able to trick me a few times in my life. Do I do this? I try not to and I don't even think I'm any good at it. Nonetheless, such questions clearly contribute to showing that I'm manipulative.

For an other example, let's take the Dark Factor test. We have here questions like:

• Most people deserve respect.

My problem with this questions is that, as far as I'm aware, there are many definitions of "respect". How am I supposed to answer this question without knowing what definition of respect we're talking about?

• The innate respect that, according to many moral theories, is an indefeasible privilege of every human being?
• The respect that is defined as a person's trustworthiness or ability to perform their duty?
• The respect that stems from fearing other person?

All 3 definitions entail radically different answers on my part, including an "I don't know, I'm not strong enough in philosophy to be able to answer this" in the first case.

Or another question. "It is sometimes worth a little suffering on my part to help others in need." Is it? I believe yes. Can I? Sadly, rarely. Which of these two answers should I give?

Another category are questions like these two: "It’s wise to keep track of information that you can use against people later." and "I feel sorry if things I do upset people." Here, however, my problems with these questions border critiquing the test itself, so I wonder if I'm not falling to the trap of arrogance. Namely, I have a feeling (maybe very wrong) that this test here treats what is not necessarily immoral as if it was. Is it wise to keep track of information that can be used against people later? In most circumstances likely no. But there are situations when this course of action is both necessary and perfectly ethical, I suppose: to protect oneself from these people anticipated unwarranted attacks. Do I feel sorry if I do things that upset other people? This depends on circumstances even more. If I win a sport match I'm likely to upset my opponents: does my attempts to win a fair game make my personality have dark traits? Escaping one's mother's overprotectiveness is likely to make her even more upset: but as far as I can tell, the psychologists' theories were that it is most healthy and necessary to do this. Nonetheless, I have a feeling that a positive answer to such a question would make one's personality darker in the results.

Could someone explain to me what's the point of such questions and how to answer them?

Answer 1

Really good question - I've had this exact conversation with a student.

It's important to understand that mature scales are validated:

In the fields of psychological testing and educational testing, "validity refers to the degree to which evidence and theory support the interpretations of test scores entailed by proposed uses of tests".

An interesting example is predictive validity:

... the extent to which a score on a scale or test predicts scores on some criterion measure. ... predictive validity provides somewhat more useful data about test validity because it has greater fidelity to the real situation in which the test will be used. After all, most tests are administered to find out something about future behavior.

From the point of view of the developer of a psychometric test, what a subject might think of a question, how they interpret (or misinterpret) it, or what ambiguity it contains, is not as important as whether or not answers to the question contribute to the validity (especially predictive validity) of the test. A question that seems meaningless to you, might actually be highly predictive (at least in general) of the criterion measure.

It is not always necessary for the developer to know why a question is predictive, only that it is predictive (provided other validation criteria are also met). Scales in general are developed using the scientific method - a survey you take that contains say, 20 questions, might originally have contained over 100 questions. After testing the scale on an experimental group, researchers may discover that 20 of those questions provide 80-90% of the predictive power of the full test, and decide to shorten it. Researchers cannot really know in advance which questions work best - that's what testing is for!

Another interesting factor in psychometric testing is demand characteristics:

In research—particularly psychology—demand characteristics refers to an experimental artifact where participants form an interpretation of the experiment's purpose and unconsciously change their behavior to fit that interpretation. ... A possible reason for demand characteristics is the participant's expectation that he or she will somehow be evaluated and thus figures out a way to 'beat' the experiment to attain good scores in the alleged evaluation.

In other words, if you feel like your test score forms some evaluation of your character, then you might try to provide answers that you think will form a desirable evaluation, rather than an accurate one.

However, it is difficult to ascertain the value or purpose of a question in a test - some scales contain questions added for testing internal validity, specifically for detecting demand characteristics and other biases that may affect results. For example, a test may contain 2 identical questions worded differently. Different responses to these questions could indicate dishonesty, misinterpretation, inconsistency, or inattentiveness. Bottom line: I wouldn't worry about what the point of a question is - don't over-think it.

Answer 2

I can understand why you are perplexed. I really does seem like these sort of questions make it so that these tests can't be accurate.

If the tests have a lot of ambiguous questions, it is either going to inaccurate, or it doesn't matter how you answer. How can it not matter what you will answer? It's because they use a technique that palm readers use in Cold Reading. It's called a Rainbow Ruse. This is when a statement both reinforces a trait as well as the opposite of it. When personality tests aren't based on research, they typically use a modified version of a rainbow ruse.