Read Audio Data in Real-time over 3.5mm Stereo Cable
The middle path

Build the simple op-amp (or whatever) 70s-era color organ, but without all the LEDs. Then basically feed the organ's LED-driving lines into the Arduino's ADCs. Sampling that pre-processed/summarized data, you can define a wide variety of RGB interactions without complicated math or an EE background. The timing requirements will be well-within the Arduino's reach, unlike computing FFTs or other "difficult" processes. Besides, old/new hybrids are cool!