Understanding capacitor usage in the Sparkfun arduino
That is a decoupling capacitor for the microcontroller. Yes, it is meant to be as close to the VCC and GND pins as possible. Noise can be picked up on the trace from between the regulator and the microcontroller. It is most likely a ceramic capacitor, which has a very low ESR(equivalent series resistance) rating for good decoupling. Smaller capacitance ceramic capacitors have a lower ESR than high capacitance versions generally. The lower the ESR, the better it is at shunting high frequency noise on the VCC rail to the device.