Find p-MOS frow battery switch application
You connected Q1 drain to the wrong place. 

With no USB power, R2 pulls the gate low, turning Q1 on. Its drain then pulls the gate high, turning it (partially) off again; it settles at 0.7V drop. 

I think you want the drain connected to TEST-POINT1, then D1 prevents the gate voltage rising. (That way the Schottky diode drop only happens under USB power, where it doesn't matter).

Note also. Q1 body diode will charge the battery from USB power, which may be very bad. You may need the usual back-to-back pair of MOSFETs to provide isolation in both directions..