How is LED powered when PIN 2 is disconnected?
The pinMode(buttonPin, INPUT); line tells the board to set the buttonPin in high impedance mode.

When you disconnect the wire the state of that pin becomes unknown; usually it oscillates.
Since it is oscillating, the LED turns on and off randomly: you see it "dimly light".

The usual solution is to use a pull-up or pull-down. I suggest you to wire your buttons towards the ground, declare the pins as pinMode(buttonPin, INPUT_PULLUP); and invert the logic (i.e. buttonState = !digitalRead(buttonPin);). This way it will never oscillate again.