Why is my red LED dimmer without a resistor than with a 100 Ohm resistor?
No resistor = way too much current
Way too much current = LED is running way too hot
When an LED gets too hot, it's ability to emit light goes down.
Touch it, bet it's really warm if not hot.
Another thing, when the temp of an LED changes,  it's output wavelength changes.  It gets longer.  So your red LED may have shifted to infrared
OR
The LEDs are overloading the power supply and its responding by folding back it's output voltage
Either way,, blue behaves differently because it has a higher voltage requirement than red.   This is probably limiting the over current condition.
They behave similar at 3V instead of 5 because there is less difference compared to Vf, so they aren't getting as much over current.
.
Got an ammeter???  Measure the current you're putting through them.