Can this driver be simplified?
This wasn't mentioned in the original question: -

\$\color{red}{\text{That the LEDs couldn't be electrically disconnected from ground}}\$

Speed of operation wasn't mentioned either and neither was there any mention of the input logic circuit being inverted so, try this as a simplification: -



You will still need to ensure that the LEDs can handle 5 volts with no chance of an over-current situation.


  I am looking for a similar driver that:
  
  
  is simple (it does not have many components). \$\color{red}{\text{YES}}\$
  does not need floating signals (and ideally only one control signal). \$\color{red}{\text{YES}}\$
  does not have any big resistors in the 100 mA path to avoid losses (in voltage or in power) \$\color{red}{\text{you need to be more forthcoming about what you currently do}}\$
  is an all or nothing driver (5 V or 0 V), so the LEDs light up properly when they are ON. \$\color{red}{\text{YES}}\$
  


PMOS load switch: -



Image from Using MOSFETs in Load Switch Applications.