Qualitative scoring for responses to design fluency task Before I can ask my questions, I have to give a little background information.  I'm involved in a study using a measure of design fluency.  Subjects are shown a page full of squares, like the one shown below.  Each square contains 5 dots.  Subjects are given 60 seconds to generate as many different designs as fast as they can by connecting dots in each square.  The criteria for correct designs are that the designs must contain 4 lines and that all the lines in the design must be continuous.  There are 2 trials in the test: one with 5 black dots and one with 5 black dots and 5 white dots.  In the second trial, the extra dots are included as form of interference. 



Here are examples of correct designs produced in the first trial of the design fluency task.  Similar designs would be likely produced under the second trial.



The way this measure is usually quantified is by counting the number of correct designs produced.  However, I'm interested in looking at this task a little differently.  I want to know if members of a clinical group, say Alzheimer's disease (AD), will produce a quality of designs that is different than normal controls.

One way I thought about doing this was by looking at the frequency of designs produced.  This has been observed for tasks of verbal fluency, with patients with AD producing words of a greater frequency than controls.  The problem is that while counts of word frequency are readily available, there is no equivalent source for production on the design fluency task that I know of.

Here's how I thought about approaching this.  I'd start by taking the responses produced by a large number of controls and coming up with a way of classifying them, say 4 or 5 types of designs.  Then I'd count the number of times each of these design types occurred in the control sample and apply a weight to them so that design types with lesser frequency would have a greater weight (think inverse document frequency).  I'd then look at a smaller AD sample and a equivalent control sample matched on demographic variables.  I'd count the number of each design types produced by each subject in both groups, and multiply the number of each design type by the weight assigned to it.  The average frequency would be calculated for each subject for each trial type (the 5 dot trial and the 10 dot interference trial).  Group performance would then be compared in a repeated measures ANOVA.

There are the two questions that I have. The control sample that I have to determine the frequency of the design types is small: less than 70 subjects.  Is this too small for determining the frequency and weights of the design types?  Secondly, when comparing the clinical sample to a control sample, would it be possible to draw the small control sample from this larger control sample used to derive the frequency and the weights, or is there some reason that I should not do this?

Thanks in advance for your attention and response.  Any input would be greatly  appreciated.  I realize this question may be off topic for this site.  If you feel that's the case, would you mind suggesting a more appropriate forum.  I've previously posted this question on Cross Validated, with no replies, so I'm hopeful for better reception here.
To measure the frequencies of different patterns (do some patterns occur more frequently that others based on group) I see this as a chi-squared test of independence. If you are unfamiliar with the test, a quick example is here. For your situation, all participants would get the same placements of dots, and you would count how often each possible pattern is drawn. In this design you would have two variables: 1 being control/AD, and the other being "design" with levels of every design drawn (if there are low frequencies on the designs, check out Fisher's Exact Test).

As you mention, the only way you could get enough data for RM ANOVA is to add up all of the frequencies for all of the graphs. Not knowing the theoretical backing for this measure, are all of them designed to be equivalent? At the very least it seems that the 5 dot and the 10 dot should be separated...

All my 2 cents, but on to your real questions:

1) Your "control sample" would be, in essence, a normative group (you would be determining what the "normal" person's frequencies were). These samples tend to be quite large and drawn with very rigorous designs. Check out your standard errors of your measures and see if they are acceptable (if they are, then the representativeness of your population may be your large concern with a small normative sample).

2) Double dipping is almost always a big "no-no." Idiosyncrasies in your initial "control sample" would drop down to your "clinical sample," casting doubt on the internal validity of the study. If the classifications are made for/from one group it usually follows that, in comparing to a second group, the second group with almost always not meet perfectly with the classification. In regression, this concept is that of "overfitting the model." One way you can get around this is create your classifications with, say, half of the control group, and use the other half as the clinical group. This has two problems, though: you further reduce your sample size; and I'd cast questions on the validity of "making classifications" (especially with so small a sample). Perhaps look at making all possible connections their own level of a categorical variable (combining only reflections and rotations).