common anode LED forward voltage higher than pin
The LED with the lowest Vf is the red one, at about 2.1V. 5V-2.1V is 2.9V, so in theory it should not be a problem for the driver, which has a Voh of 3.3V (the 3.3V rail) and therefore not forward bias the LED.
In practice, what happens is that the LED has enough sub-threshold leakage current that even this minuscule amount will cause it light dimly, sinking current into the I/O pin protection circuit. The current is small enough that it isn't really harmful to the driver, but leakage is leakage and you don't want that.
There's also robustness to power sequence to consider too: if your 3.3V supply is off and 5V is on, leakage current can get through the LEDs to the driver and phantom-power the 3.3V parts of the circuit. This is never a good thing and is to be avoided.
Instead, consider driving the LED using an open-drain buffer like an LVC07 or discrete FETs like 2N7001 or similar. Then there's no possibility of leakage or phantom-power and you'll have plenty of current to drive the LEDs as brightly as you want.