Hacking a ball mouse with Arduino
The output of the optical sensor in a mouse like this is a digital signal. It should swing between the mouse GND and the voltage equal to the VCC pin of the optical sensor.

To have your Arduino take the place of the optical sensor the Arduino should be driving the signals to the mouse's controller chip with digital output pins. The voltage swing between a '0' level and the '1' level should closely match that which the optical sensor was producing.