MOSFET reduces LED switching time - why and how to resolve
Instead of a MOSFET consider a fast CMOS logic gate as a LED driver. For example AC04 with all gates in parallel. You will get low output impedance, low input capacitance and short switching times. Please use good decoupling and short connections - the circuit is plenty fast.