Arduino - pin13 is HIGH by default
Taking a look at the Arduino Uno's schematic explains this, if I understand the schematic correctly. I don't own an Arduino Uno, so I can't verify with a multimeter, so I might be wrong.

As I understand from the schematic, pin 13 of the arduino, the SCK pin, is connected to an op amp buffer. When you (or anyone) upload a program that doesn't do anything to pin 13, it's default state is an input without a pullup. 

When a digital pin is configured as an input without a pullup resistor, it has a high impedance (high-Z) and unspecified voltage. It can be 0V, but might be something like 1.5V as well. The measurement will depend on the impedance of your meter, among other things. Upload an empty sketch and check the voltage on the pins - you'll find they're pretty random and not always 0V.

Now, a high-Z input pin with, say, a few volts will not be able to power an LED. However, this voltage is buffered by the Uno's op amp, and repeated on the output. This op amp will be able to power up an LED, and apperently, it does.

I think the Arduino team should add a high ohmage pulldown resistor (e.g. 50kOhm) to make this less confusing.