4
$\begingroup$

I've results of two tests administered to university students:

  1. the 18-item true-false version of the Self-Monitoring Scale (Snyder & Gangestad, 1986);
  2. the Raven Progressive Matrices Test (John & Raven, 2003).

In the past I've been assessing personality profiles of students, referring to the five factor model of personality and using a test based on the International Personality Item Pool (Goldberg et al., 2006). In this case I was measuring reliability by Alpha coefficients. However, these coefficient seems not to make sense in a test like Raven's where there are only correct and wrong answers. Same doubts apply to the other test.

Should I measure the reliability of these two tests (Raven and Self-monitoring) on my sample? If so, which measure should I use?

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
1
$\begingroup$

Regarding Cronbach's $\alpha$, Wikipedia writes:

It has been proposed that $\alpha$ can be viewed as the expected correlation of two tests that measure the same construct.

Note that correlation is a nonparametric concept. Its definition is independent on the functional form and functional relationship of the two variables. There are numerous ways to measure correlation that take the functional form into account. Pearson correlation and point-biserial correlation are probably the most well known.

In the case of two binary variables some notable measures are: Matthews correlation coefficient, Cohen's $\kappa$ or Pearson's $\phi$. One may also consider measures from Signal detection theory. Cronbach's $\alpha$ is also applicable to binary variables. In fact, the Wikipedia page lists a simplified formula (due to Cronbach, 1970) for the case where both variables are binary.

To conclude, there is nothing wrong with application of Cronbach's $\alpha$ to binary items. Still, other measures of relationship between two binary variables exist and may be worth considering.

Reference: Cronbach LJ (1970). Essentials of Psychological Testing. Harper & Row. p. 161.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks! So this means that Alpha, or for instance its alternative Kuder–Richardson Formula 20, are ok for reliabilty of the Self Monitoring Test, right? There items are binary, true/false. But what about the Raven where respondents are not expressing their level of accordance to the construct, but they are just looking for the correct answer? $\endgroup$
    – Forinstance
    Jul 24, 2015 at 7:27
  • 1
    $\begingroup$ The theoretical construct behind Raven is the reasoning ability. It's not necessary that the subject understands what is being measured and what is the theory behind the questionnaire design. They just need to understand the questions posed by the test and what their task is. In fact in some cases it may be desirable to use implicit measure. With explicit measures the subject may fabricate responses in order to obtain some desirable score. $\endgroup$
    – matus
    Jul 24, 2015 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.