What causes student errors on a problem involving differentiating a factorial? Consider the following maths problem, assuming only a high-school level knowledge of calculus: 


  If $f(x) = x!$, find $df/dx$ 


Almost all of my respondents seemed to switch on their "mental differentiation pipeline" and tried to expand $x!$ as a product and got stuck. The right answer is that $x!$ is undefined for any non-integer, so $f$ is discontinuous, and $f'$ doesn't exist!

Is there a name in psychology for the phenomenon that causes one to think along the wrong lines in the problem above?

Is it the same as the one leading to incorrect answers to the bat and ball problem? Is this a test of your thinking style? For, perhaps an intuitive thinking style will  lead one to expand $x!$ as a product and attempt to differentiate term-wise. Is there a "cognitive remedy" for wrong thinking?
I imagine this question is tricky for students for a several reasons. 


Question wording: The question may suggest to the student that it is possible to differentiate $x!$. Or they may assume from the wording that some meaning is meant where it is possible to differentiate. For example, alternatively, you could ask a set of questions, one for each functions, about whether a given function was differentiable (e.g., $x^2$, $2x+3x^3$, $log(x)$, $x!$). This would clearly be easier, because it puts the potential that the function might not be differentiable into the student's mind. I'm not saying what method of questioning is better from a pedagogy perspective; I'm just noting that question wording influences the difficulty of the problem, and that in this case, the question might be labelled a "trick question".
Analogy to solve novel problems: If students think that there is an actual derivative as an answer, but no obvious solution exists, it is natural to apply strategies that seem relevant to the problem. For example, they might know how to calculate $x!$ for any given positive discrete value of $x$. Therefore they might assume that working out the derivative involves interpolating (e.g., something like a gamma function) or they might rely on what they know about the product rule or some such. In general it is common for people to try to solve problems that they don't understand by drawing on strategies that seems relevant.
The question relies on the ability to apply a key fact that discrete functions are not differentiable. Thus, students need to know that fact, and they need to know that the factorial is a discrete function, and they need to link these two ideas. Thus, from a knowledge perspective, this question relies on a piece of knowledge that students might not possess.


So, in summary, I think the question exemplifies the kinds of challenges that students encounter when learning and demonstrating mathematical skills. I think a lot can be learnt about students cognitive representations of a problem by the kinds of errors they make. However, I do not think failing to solve this problem correctly says anything fundamental about your thinking style.

For further discussion of cognitive science in relation to mathematics education, perhaps check out Robert Siegler's work.

References


Siegler, R. S. (2003). Implications of cognitive science research for mathematics education. In Kilpatrick, J.,
Martin, W. B., &amp; Schifter, D. E. (Eds.), A research companion to principles and standards for school. mathematics (pp. 219-233). Reston, VA: National Council of Teachers of Mathmatics.
PDF