Ballpark achievable switching frequency for 100mW D-class amplifier
Assuming 200mW peak power delivered into 8 ohm load, that gives peak voltage and current of 1.26V and 15mA. V peak to peak required is 2.52V. This is within the realm of logic output with no FET at all, perhaps even directly off of the FPGA, or a simple buffer. Have I forgotten to consider something?

If FETs are used, given the low power level, it does not appear any "special" drivers are needed as long as the drive signals have the correct voltage levels. The switching frequency would be limited by concerns which some you have mentioned and few MHz should be simple. Also, at the low power level, some datasheet values are likely to be quite conservative because they are specified at relatively high voltage and current.