Before I can ask my questions, I have to give a little background information. I'm involved in a study using a measure of design fluency. Subjects are shown a page full of squares, like the one shown below. Each square contains 5 dots. Subjects are given 60 seconds to generate as many different designs as fast as they can by connecting dots in each square. The criteria for correct designs are that the designs must contain 4 lines and that all the lines in the design must be continuous. There are 2 trials in the test: one with 5 black dots and one with 5 black dots and 5 white dots. In the second trial, the extra dots are included as form of interference.
Here are examples of correct designs produced in the first trial of the design fluency task. Similar designs would be likely produced under the second trial.
The way this measure is usually quantified is by counting the number of correct designs produced. However, I'm interested in looking at this task a little differently. I want to know if members of a clinical group, say Alzheimer's disease (AD), will produce a quality of designs that is different than normal controls.
One way I thought about doing this was by looking at the frequency of designs produced. This has been observed for tasks of verbal fluency, with patients with AD producing words of a greater frequency than controls. The problem is that while counts of word frequency are readily available, there is no equivalent source for production on the design fluency task that I know of.
Here's how I thought about approaching this. I'd start by taking the responses produced by a large number of controls and coming up with a way of classifying them, say 4 or 5 types of designs. Then I'd count the number of times each of these design types occurred in the control sample and apply a weight to them so that design types with lesser frequency would have a greater weight (think inverse document frequency). I'd then look at a smaller AD sample and a equivalent control sample matched on demographic variables. I'd count the number of each design types produced by each subject in both groups, and multiply the number of each design type by the weight assigned to it. The average frequency would be calculated for each subject for each trial type (the 5 dot trial and the 10 dot interference trial). Group performance would then be compared in a repeated measures ANOVA.
There are the two questions that I have. The control sample that I have to determine the frequency of the design types is small: less than 70 subjects. Is this too small for determining the frequency and weights of the design types? Secondly, when comparing the clinical sample to a control sample, would it be possible to draw the small control sample from this larger control sample used to derive the frequency and the weights, or is there some reason that I should not do this?
Thanks in advance for your attention and response. Any input would be greatly appreciated. I realize this question may be off topic for this site. If you feel that's the case, would you mind suggesting a more appropriate forum. I've previously posted this question on Cross Validated, with no replies, so I'm hopeful for better reception here.