On my Arduino, why is this LED always on, even though I told it to turn off?
I believe your problem is that your not setting that pin as an output. Use pinMode(13, OUTPUT) to configure the pin 13 to be used as a digital output. Since GPIO pins can be used as a input or outputs on/off, you need to let the micro controller know what mode that pin needs to be set to.

#include &lt;EEPROM.h&gt;
int led = 13;

// the setup routine runs once when you press reset:
void setup() 
{                
  // initialize the digital pin as an output.
  pinMode(led, OUTPUT);

  // write a 0 to all 512 bytes of the EEPROM
  for (int i = 0; i &lt; 512; i++)
    EEPROM.write(i, 0);

  digitalWrite(led, HIGH);   // turn the LED on (HIGH is the voltage level)
}

void loop() 
{
}


You can see the basic Hello World example here, that goes over blinking the pin 13 LED.

Also, you should be careful about writing to the EEPROM, as pointed out in the comments, it only has ~100k cycle lifetime.