Over heating H-bridge MOSFET (irf9540)?
There are several errors in your Proteus simulation:-


The LOG pots you have chosen for RV1, RV2 and RV3 do not appear to work. After I switched RV1 to POT-LINthe 555 timer started generating a PWM signal.
Using 100k potentiometers to change direction is not appropriate. RV2 and RV3 should be changed to a DPDT switch or digital signals.
There is no connection between Q5 and Q3 or Q11 and Q10 (they only look connected because they are placed too close together). You need to move them apart, then add a wire from the Emitter of the upper transistor (Q5/Q11) to the Collector of the lower one (Q3/Q10).