negative voltage appears on PNP transistor collector
It's not normal. When the transistor is 'off' it is effectively an open circuit save for a bit of leakage to the +5V rail. A normal meter provides a an resistance of maybe 10M or 10G+ in some cases to the 0V rail. In any case, the voltage should be between 0V and 5V (and most likely close to 0V). 

If you have a negative supply somewhere in your circuit, maybe there is a resistance between that and the PNP collector, if not then maybe something is up with your meter- it should read 0V when the test leads are open.