Getting gibberish serial port until button is pressed, plus the button is turning the leds off instead of on
When pin 2 is connected to ground as you have done it puts the chip into an invalid boot mode. It will be spewing out messages at 74880 baud complaining about the boot mode being invalid.

You must not connect pin 2 to ground. It must, at boot time, be connected to 3.3V through a resistor (which is should be provided by the board for you if you use a development board not a bare module).

By connecting the button between 3.3V and GND you are causing a short circuit that removes the power from the module and risks damaging the power supply on your board.

The proper way of wiring a button is to use:

pinMode(2, INPUT_PULLUP);


and then connect the GPIO pin to ground through the button.

The INPUT_PULLUP acts like the required pullup resistor keeping the input HIGH while "idle", and the button connects the input to GND when you want to "activate" it, making it LOW.

Those are the only connections you should make.  If you want to use an external resistor instead of the internal one with INPUT_PULLUP, then that goes between the input pin and 3.3V.

Never ever cause 3.3V to be directly connected to GND.