Why so few subjects in psychophysics? Why is it that most psychophysics experiments only use a handful of subjects (typically around five, but sometimes even four, three or two), whereas the "norm" for experimental psychology (assuming a medium effect size) is around N=25? Is the motivation to reduce sampling error and increase external validity not the same for psychophysics as it is for e.g. decision-making experiments? Or is it the case that individual differences are smaller in perception than they are in cognition?
It is partly as you already guessed, because the phenomena that are studied don't vary as much between individuals. This doesn't necessarily mean that individual differences in perception are smaller, it means that individual differences in those aspects of perception that are studied in psychophysics are smaller.

Another reason is that it is customary in psychophysics to instead use large numbers of stimuli - sometimes thousands - thus making sure that the estimate per individual is very precise. In that sense the more cognitive topics are much more underpowered, often employing groups of several tens of stimuli to generalize about entire classes of objects.