Is my LED driver fundamentally incorrect, or can I compensate it somehow?
That is a pretty weird way of &quot;driving&quot; LEDs.
OP's schematic:

A few issues:

essentially you have an &quot;equivalent&quot; resistor, shared by the LEDs (if the LEDs have any imbalance - which they have - the current draw will be uneven)
power draw - when the LEDs are &quot;off&quot;, the FET will be drawing the current (even more than the amount drawn by the LED since there will be no V_LED)

The &quot;regular&quot; way of doing it would be (for each of the tricolor LEDs, then you can tie the GPIOX to whatever color combination you need):
(Assuming you dont need level shifting. Gate/Pull-up resistors omitted.)



simulate this circuit &ndash; Schematic created using CircuitLab