What happens when I exceed amp limit on Arduino 5v pin?
If you exceed the amp limit, that can be drawn through (!) the Arduino, some components on the Arduino will be fried (As for most electronic devices. For the Arduino this is practically a short). There is no current limitation there. Which components are fried depends on how you provide the power. If you provide power through the power jack or VIN, the voltage regulator will burn out. If you provide power through USB a diode or the MOSFET will be fried (You can look this up in the Arduino UNOs schematic at this site).

The setup might output a current, that is a bit higher than 500mA, for a short time. But the mentioned components, that are in the currents path, will heat up very quickly and be destroyed. In this short time the LEDs will be very dim, since the Arduino cannot provide a current as big as 8A, not even for a short time.

All great loads have to be powered separately from the Arduino, so that their supply current will not flow through the Arduino. If you are using a powerful USB supply, you have to split the power lines, before giving the voltage to the Arduino. You can connect the USBs 5V pin to the 5V pin of the Arduino and to the LEDs. If you are using a power supply with a higher voltage than possible for your LEDs, you have to buy an extra regulator, that also capable of providing the needed current.

Important note: Since you didn't specify, how you are connecting the RGB LEDs, you also have to take care about the current limit of a single pin. It should not exceed 20mA, 40mA at the absolute maximum. So you cannot drive the LEDs directly. You will need a driver for then anyway. The simplest way is a MOSFET, that is capable of sourcing the corresponding current. It is then controlled by the Arduino, which needs very little current for it.