Calculate threshold from 'binary' yes/no data For various reasons, I have a crappy experimental design that I am forced to deal with. I present subjects with 20 yes/no trials, out of which 5 are no-signal catch trials and 15 are signal trials. Each signal presentation has a unique level with unequal spacing (as small as one in arbitrary units and as great as 5). All subjects run the trials in the same order (i.e., the 5th trial is always at signal level, and the 11th and the 6th trial are always catch trials).

To make things more complicated, at the population level, and likely at the individual level, the criterion is not stable such that the probability of saying yes on a no-signal trial depends on the catch trial number (the 1st catch trial has a probability of yes of about 0.05 and the 5th catch trial has a probability of yes of about 0.2). From this mess, I need a single number representing threshold. I don't really care what the threshold corresponds to (e.g., the signal level leading to 50% yes, 50% correct, or a d' of 1 would all be fine). Ideally, the method would provide a confidence interval on the estimate at the individual subject level.

I was thinking of using a Spearman-Karber type approach, but the unequal step sizes seems problematic (along with the variable criteria). I also don't understand the history of Spearman-Karber (i.e., where was it first published and how it relates to Spearman 1908) and kind of hope that maybe in the past 100 years we have improved on things.
The biggest drawback of your procedure, also point of discussion previously here is the limited number of trials. Apparently, you obtained binary data, namely one answer (yes/no) per trial, basically leaving you with a sparsely sampled, binary data set. Binary data will not allow a curve fitting, which is the conventional way to determine a threshold.

To make matters worse, there is a certain bias in the catch trials. But values anywhere below 25% false positive are, imo fine. So let's forget that issue, as it is likely the least of your problems.

Now, single-stimulus per trial data are referred to as the Method of Limits. Basically what you would normally do to obtain a threshold with MoL is increase the level to see where the subjects starts perceiving the stimulus (YES/NO reversal) and then flip the sequence and see where the person stops answering YES and hits NO. Generally, such a procedure is experimentally used to get a quick handle of a good starting point for the 'real deal' experiments. Or, it is useful in clinical environments. For example, clinical audiometric thresholds are often determined this way - quick and dirty, but accurate by about 5 dB or so.

The response you get in this case is a so-called type 2 response, since you never can tell whether the subject was right or not. You just evaluate when the subject reports to perceive the stimulus. The problem with the method described here is that the S may become habituated (I didn't perceive it now, likely I won't in the next trial) or the S makes the error of expectation (It will probably become apparent now). 


The only way in which I can see to analyze your data is determine the flipping point and take that as a measure. You can then, perhaps take the average between the two intensities where the response flips to get an estimate of the 50% level, the gold standard of the threshold in yes/no tasks.
Likely you will not have an unambiguous flipping point in your data, but I'm sure you can think of something. Perhaps just take the lowest intensity where a flip occurred, or something along those lines.      
Alternatively, you may group your intensities into approximate levels and average out the responses to get values including rates between 0 and 100%. This procedure, however, is likely pretty much doomed to be shot to pieces by any referee or peer in your field, the more since your step sizes are unequal. 


Reference
- Kingdom &amp; Prins. Psychophysics. Elsevier 2010