Mosfet is getting hot in switching circuit
There are two important types of losses in a mosfet: Conduction losses and switching losses.


Conduction losses: This occurs when the mosfet is already switched on and is conducting current. You will have some resistance from drain to source. This resistance changes with the gate voltage. In this circuit, when Vch is pulled low, Vgs is about -4V. The resistance Rds will cause a small voltage drop. At 0.75A, for the Si2351DS you'll get Vds=0.075V, but for the IRF9540N it will be closer to Vds=0.4V. You get 0.3W loss just from conduction losses.
Switching losses: The gate of a mosfet acts as a small capacitor. While the gate charges, some current can flow trough the device, but there will still be a voltage drop across it. This is demonstrated in the following graph.

Switching losses also occur when switching the mosfet off.
For Q1: lets look at the input capacitance of the Si2351DS vs the IRF9540N, which is 250 vs 1300 pF respectively. It will take 5 times longer to charge the gate of the IRF9540N, so the switching losses increase. The same goes for Q3: 2sk3539 vs IRLB8721PbF (1077 vs 12 pF respectively.) In addition to this the IRLB8721PbF is fully switched on at a Vds of 6V whereas 2sk3539 is only requires 3V. This is important, because Q3 is used to accelerate the charging of the gate of Q1.


A good solution would be to use the same mosfets as used in the reference design or something comparable, not just any mosfet.

I hope this helps!

-kv