Why do R and C have to be small for differentiator circuit
If you rearrange the first equation, you get

\$C\dfrac{dV_{in}}{dt} = \dfrac{V}{R} + C\dfrac{dV}{dt}\$

So if you reduce C enough, you'll make the derivative term on the right insignificant compared to V/R, and you'll get your second equation.

Alternately, if you reduce R, you'll make the V/R term larger, and again the right-hand-side derivative term will become insignificant, and you'll get the desired result.

So I'd say it's not that you must decrease R and C together, but you do have do some combination of reducing R and reducing C to make the circuit work like a differentiator.