Clothes dryer exhaust duct touching PVC pipe
The glass temperature (point at which it starts becoming molten) for PVC is 82°C (179°F). The generally accepted maximum operating temperature for PVC is 70°C (158°F) though again, this can vary by product. Pressure ratings are at room temperature (21°C/70°F), and they go down as the temperature goes up: at 70°C the pressure rating is decreased by around 20% (eg, schedule 80 PVC rated for 230psi is only rated for 43psi at 70°C). 

(Note: CPVC, which is more commonly found used for potable water applications, has a glass temperature of 106°C (222°F) and unlike PVC, it can also be used for hot water supplies. Since you said drain pipe, it's probably regular PVC.)

The exhaust temperature of a dryer typically cycles from 50 to 71°C (120 to 160°F), though there is a potential if the dryer is broken it could be higher (any dryer made since the 1980's has a thermal fuse in it, hard to find info but it sounds like these often blow at 198 or 185°F). 

So worst case, is if the pipe was a pressurized PVC water pipe, and you have really high water pressure (90psi -- typical is 30 to 60psi), and your dryer is broken and operates near 198°F for a while before the safety fuse shuts it off: depending on how enclosed this area is, it may actually get the pipe hot enough to weaken it enough that it bursts from pressure. 

However, drains are not pressurized, and so in the same situation, the temperature is still not high enough to get to the glass temperature, and from 4" away, it's definitely not going to transfer that much heat to the PVC (unless you had it enclosed in a small insulated space with no air flow at all). I'd say you are pretty safe.