Boost converter as power supply for H-bridge
I noticed that all your mosfets are N-channel.

It would be easier to use P-channel mosfets for M2 and M3, since the Vgs is more stable than for NMOS.





I think the behavior you're seeing in the waveforms is because Vgs is varying with the current through R1.

From http://www.learn-c.com/experiment7.htm:




  There is a bit of a problem when a MOSFET is used for Switch A or
  Switch B. The Drain is connected to the V+ for the motor and the
  Source to the motor. If the gate is then made to be 2 volts greater
  than the Source, the MOSFET will turn on. When that happens however,
  the voltage at the Source will increase until the gate is no longer 2
  volts greater than the Source and the MOSFET will turn off. Another
  voltage source is needed that is always greater than Motor V+ by at
  least 2 volts.