How to compare linear and categorical models of perceptual comparisons? Say I have a confusion matrix of perceptual similarity data for stimuli drawn from an underlying continuous spectrum. Participants also learned to categorise these stimuli such that the spectrum is evenly divided into two categories. I want to test whether the observed confusion matrix shows a better fit to either a continuous or a categorical model.
One approach I have considered is to create a hypothetical confusion matrix for linear and categorical models (see below) and correlate these with the observed confusion matrix.



Note: Similarity values, 1 = identical, 0 = dissimilar

However, I’m not sure how to compare the degree of fit for each model. One idea is to take the partial correlation to measure how well the confusion matrix correlates with the categorical model after the correlation between the linear model and the data has been accounted for. This approach seems to have good psychophysical validity (since any categorical process would function in addition to a linear sensory one), however I’m concerned that the linear and categorical models are highly correlated (here at 0.83) and so any partial correlation approach would not be valid. Including more stimuli near the category boundary would probably go some way to decorrelating these models, however I’m wondering if there is a more sensible approach to figuring this out (but not orthogonalising the models)? Any leads appreciated.
This sounds a lot like tests of categorical perception done in psycholinguistics. Categorical perception states that people perceive phonemes categorically, even though the acoustic signal varies continuously.
For example, the phonemes $/b/$ and $/p/$ differ on only one acoustic feature: voice onset time. Using computer synthesized speech, one can experimentally manipulate the voice onset time along a continuous dimension. In a prototypical experiment, subjects are asked to identify each sound as a $/b/$ or $/p/$. If perception were continuous, you might expect that the proportion of $/b/$ responses varies linearly with the voice onset time-- but it doesn't. As you can see in the red line, there is a sharp drop around 20ms in which people perceive everything to the left as $/b/$ and everything to the right as $/p/$.

Another way to see this is to plot discriminability (blue line). Here, participants are presented two stimuli from adjacent VOT bins and asked to judge which one matches a 'target' stimulus. Generally, people perform near chance at discriminating between the two phonemes unless the two stimuli cross the VOT boundary. Again, if perception were linear, you might expect a straight blue line rather than a peaked one.
Most researchers will use t-tests or ANOVA to test for differences of between and within category discrimination. For instance, one might conduct a series of t-tests on discriminability to show that only when stimuli cross the VOT boundary can subjects reliably discriminate them.
For an example of these tests in practice, see Morford et al. (2008). Note that some researchers have found conflicting results when using different dependent measures. For example, Massaro &amp; Cohen (1983) asked participants to make continuous judgments of stimuli and found no evidence of categorical perception.
Another alternative might be to find the best fitting line and the best fitting sigmoid curve, and compute the Bayesian Information Criterion for each. The one with the lower BIC is the preferred model. For regression analysis, BIC can be calculated as

$BIC = n\log(\frac{RSS}{n}) + p\log(n)$

where n is the number of data points in the model, p is the number of free parameters, and RSS is the residual sum of squares.
The drawback of using BIC is that it only tells you something about comparative model fit. If both linear and sigmoidal models were a bad fit for the data, BIC would not tell you that.

Massaro, D. W., &amp; Cohen, M. M. (1983). Categorical or continuous speech perception: A new test. Speech Communication, 2(1), 15-35. PDF
Morford, J. P., Grieve-Smith, A. B., MacFarlane, J., Staley, J., &amp; Waters, G. (2008). Effects of language experience on the perception of American Sign Language. Cognition, 109(1), 41-53. PDF