Design of output stage for CNY70 reflective sensor
I believe what you're missing is the collector current vs. collector to emitter voltage for given values of diode current.  See the curves I copied from your datasheet that you referenced above.



You're trying to obtain 5 volts output at the emitter, using Vcc=5V.  This leaves no or very little voltage for VCE.  Looking at the curves below, for IF=20mA, the collector current drops off rapidly once VCE falls below about 0.3V.  In effect, the CTR is no longer 5% at this low value of VCE.

In the case of your first set of values, your 5k emitter resistor with 1.5V developed across it indicates a current of 300uA and a VCE of 3.5V.  This is consistent with what is shown in the curves.