How does sin/cos encoder increase the resolution of an incremental encoder?
Very simply this is the difference between analog and digital.

The digital accuracy (number of steps per turn of the encoder) is fixed by the number of changes that can be sensed. The greater the number of steps / lines the finer the resolution. This is determined by the number of digital sensors and or lines on the optical plate the sensors use. This is fixed. If you want more increments you have to get a better encoder.

The analog sin/cos encoder uses 3 coils (one driver coil and 2 sensing coils).
The sensing of the signals uses an ADC. By using a better ADC you can "increase" the number of steps / lines per turn by increasing the ADC resolution. The encoder signal is continuous and Not Digital. So in theory you can use better measurements to get better resolution. (Noise in the measurements limits this in the real world)  

Interpolation is a method to determine a value based on an known "calibration" values (Say you know 0 deg and 90 deg for the 2 sensing coils) you can then work out any value in between by using some simple "interpolation math" between the know "calibration values"

http://www.ti.com/lit/ug/tidua05a/tidua05a.pdf



So by looking at the relative signals you can "interpolate" more lines (finer resolution) if you have good ADC's because you know the relation ship between signals.