Qualitative scoring for responses to design fluency task

Question

Before I can ask my questions, I have to give a little background information. I'm involved in a study using a measure of design fluency. Subjects are shown a page full of squares, like the one shown below. Each square contains 5 dots. Subjects are given 60 seconds to generate as many different designs as fast as they can by connecting dots in each square. The criteria for correct designs are that the designs must contain 4 lines and that all the lines in the design must be continuous. There are 2 trials in the test: one with 5 black dots and one with 5 black dots and 5 white dots. In the second trial, the extra dots are included as form of interference.

Here are examples of correct designs produced in the first trial of the design fluency task. Similar designs would be likely produced under the second trial.

The way this measure is usually quantified is by counting the number of correct designs produced. However, I'm interested in looking at this task a little differently. I want to know if members of a clinical group, say Alzheimer's disease (AD), will produce a quality of designs that is different than normal controls.

One way I thought about doing this was by looking at the frequency of designs produced. This has been observed for tasks of verbal fluency, with patients with AD producing words of a greater frequency than controls. The problem is that while counts of word frequency are readily available, there is no equivalent source for production on the design fluency task that I know of.

Here's how I thought about approaching this. I'd start by taking the responses produced by a large number of controls and coming up with a way of classifying them, say 4 or 5 types of designs. Then I'd count the number of times each of these design types occurred in the control sample and apply a weight to them so that design types with lesser frequency would have a greater weight (think inverse document frequency). I'd then look at a smaller AD sample and a equivalent control sample matched on demographic variables. I'd count the number of each design types produced by each subject in both groups, and multiply the number of each design type by the weight assigned to it. The average frequency would be calculated for each subject for each trial type (the 5 dot trial and the 10 dot interference trial). Group performance would then be compared in a repeated measures ANOVA.

There are the two questions that I have. The control sample that I have to determine the frequency of the design types is small: less than 70 subjects. Is this too small for determining the frequency and weights of the design types? Secondly, when comparing the clinical sample to a control sample, would it be possible to draw the small control sample from this larger control sample used to derive the frequency and the weights, or is there some reason that I should not do this?

Thanks in advance for your attention and response. Any input would be greatly appreciated. I realize this question may be off topic for this site. If you feel that's the case, would you mind suggesting a more appropriate forum. I've previously posted this question on Cross Validated, with no replies, so I'm hopeful for better reception here.

Answer 1

To measure the frequencies of different patterns (do some patterns occur more frequently that others based on group) I see this as a chi-squared test of independence. If you are unfamiliar with the test, a quick example is here. For your situation, all participants would get the same placements of dots, and you would count how often each possible pattern is drawn. In this design you would have two variables: 1 being control/AD, and the other being "design" with levels of every design drawn (if there are low frequencies on the designs, check out Fisher's Exact Test).

As you mention, the only way you could get enough data for RM ANOVA is to add up all of the frequencies for all of the graphs. Not knowing the theoretical backing for this measure, are all of them designed to be equivalent? At the very least it seems that the 5 dot and the 10 dot should be separated...

All my 2 cents, but on to your real questions:

1) Your "control sample" would be, in essence, a normative group (you would be determining what the "normal" person's frequencies were). These samples tend to be quite large and drawn with very rigorous designs. Check out your standard errors of your measures and see if they are acceptable (if they are, then the representativeness of your population may be your large concern with a small normative sample).

2) Double dipping is almost always a big "no-no." Idiosyncrasies in your initial "control sample" would drop down to your "clinical sample," casting doubt on the internal validity of the study. If the classifications are made for/from one group it usually follows that, in comparing to a second group, the second group with almost always not meet perfectly with the classification. In regression, this concept is that of "overfitting the model." One way you can get around this is create your classifications with, say, half of the control group, and use the other half as the clinical group. This has two problems, though: you further reduce your sample size; and I'd cast questions on the validity of "making classifications" (especially with so small a sample). Perhaps look at making all possible connections their own level of a categorical variable (combining only reflections and rotations).

Answer 2

This idea of score is interesting, but it's painful to assess if the two problems you raised are important or not. For the re-use of the "meaure group" I think it would be careful to not do it. I took much thinking over it and I still don't know what to think. Unfortunately, I have no solution, but if I share my different thoughts process here maybe it would help.

1] I first thought that you should better build a rank of your different categories than to build a measure to avoid the problems you raised.

It would need not much subjects to get a reliable result. If you build a rank, you can defend the idea of using the same group for the two phases because it would have been extremely unlucky to get a different rank using a different sample for the same population. It can be assessed statistically how unlucky it would have been.

I must point out a problem that you have to deal with. Let's imagine two different populations one has this rank $A>B>C>D$ the other this one $A<B<C<D$. Unless you have some a priori to the population, you can't judge than one is more "creative" than another. I known that "creative" is my words not yours but one must be careful about the interpretation of the different distribution you will have. If you wan't to avoid this, no need to build a rank for this reason, and just go for an independance test as mfloren suggested.

Let's say that you still conserve this rank.

To test your hypothesis, I think the simplest way is to compare distributions of creations between your two groups. ordered categories with a 1-tailed kolmogorov-smirnov test because it's directly related to the hypothesis of you want to test. It is based on the biggest difference in the cumulative distribution of your two groups across you ordered category.

But there is a big problem (even for the case of independance test) : One could object that the i.i.d assumption is not met because subjects give many answers and some might be more creative than others. I don't know any between+within subjects alternative to kolmogorov smirnov. The simplest way to avoid this problem is to get back to 1 subject=1 data point...and this is where I am stuck because I don't see a nice way to pick one category. I must say that with your measure, this problem do not occur.
2] I am not sure that avoiding the scoring solution was a good idea because of this pesty i.i.d problem.

Answer 3

This is a follow-up after my first answer, this is based on the suggestion of @mfloren in the comments part of his answer.

You can't really use an independence test or a kolmogorov smirnov test because your data are not i.i.d. (your subjects give more than one answer). Your scoring idea would fix this, because you reestablish 1data=1subject but this has two problems :

• Subject-consuming (one group to establish the measure, and 2 groups to compare). The idea to use the same subjects for the measure and for the control group is generally a thing to avoid even if I am not 100% sure that it causes big trouble here.
• If your design, under certain conditions (reverted rank for example) AD could be shown as "more creative" than healthy subject, but the contrary could also be true if you use AD patients to build your scoring system. Using healthy people has a reference point is maybe not a big deal, but this is an argument which deserves to be dealt with.

A simpler measure which would avoid these problems would be to consider the number of different design produced. From 0 to 6 if you have 6 different designs then. This is the proposal of @mfloren in the comments. I think it deserves a proper answer.

As you have two conditions, you can use a Friedman test which is a sort of non parametric ANOVA and you may use wilcoxon tests to study each condition with a multiple test correction like Bonferroni-Holmes.