Is the first derivative operation on a signal a causal system? Please help me sort this issue out. 

Consider a system whose output $y(t)$ is the first derivative of the input signal $x(t)$.

We can write the first derivative of an input signal as follows:

$$y(t)=\frac{dx(t)}{dt}=\lim_{\Delta t\to 0}\frac{x(t+\Delta t)-x(t)}{\Delta t}$$

Thus, we can see that the system's output must remember an infinitesimal change in future input in order to calculate the derivative, thus system possesses memory. Perfectly fine till here.

The problem comes in checking if it's causal. By the definition of the derivative above, the system has to subtract an infinitesimal future value with the current value, to calculate the derivative. Thus by definition, this has to be a non-causal system, right?

However, most of the textbooks and websites mention the first derivative as a causal system. 

Please let me know if I am making any mistakes. Thank you.
If the derivative exists at the given point, then it doesn't matter if you look (infinitesimally) into the future or into the past, you can do both, because both will give the same result:

$$x'(t)=\lim_{h\to 0}\frac{x(t+h)-x(t)}{h}=\lim_{h\to 0}\frac{x(t)-x(t-h)}{h}\tag{1}$$

So a differentiator can be (theoretically) implemented by a causal system.

Clearly, the system is not stable, because a bounded input can cause an unbounded output. E.g., a step at the input causes an unbounded output signal. So if by realizable we understand causal and stable, the ideal differentiator is not realizable. However, good approximations in a certain frequency range can be realized with small errors, in the continuous domain as well as in the discrete domain.