BLDC controller MOSFET heating problem
The bootstrap circuits in the mosfet driver expect the control signal frequencies to be at least many tens of kHz. Looking at your waveforms gives me the impression that your control signals are more likely tens of Hz and that is your problem. 

Bootstrapping requires a fast changing voltage on the mosfet outputs else it fails to work correctly. An alternative is to use p channel mosfets in the upper position or, add a supply that is about 10 volts higher than Vbat.

It’s a bad design if you are expecting this to work without continuous signal changes in the mid kHz region and thinking that the circuit can be used as a stepper controller at low speed is your mistake.