Attiny85, reading analog values from PIN3, reports 654 on idle?
As @microtherion has said, there are components on PB3. Mainly, a pull-up to 5V, a pull-down 3.6V Zener Diode, and a current limiting series resistor. This is for the USB D- line. PB4 has just the Zener and a series resistor, for the USB D+ line. These components are to keep the USB lines within voltage specs for the Data lines (which are not 5V, that's just USB power), and to signal to whatever host that it connects to that there is a USB device inserted.
Considering the ADC is set to full voltage range of 5V, with 0V corresponding to a result of 0, and 5V a result of 1023, a result of 654 is roughly 3.2V sensed at the Analog Input.
As the line is connected to a 3.6V Zener (or maybe 3.3V on your specific board), this explains your result. Especially if you are using the USB feature at the same time. Conflict.
Once you moved to a free pin, you just need to wire it up correctly.
Pin 2 or 4 of the updated board schematic needs to be tied to ground. And then enable the input pull-up resistor, or add your own, from Pin 1 or Pin 3. The internal pull-up is typically 47k on most MCUs.



simulate this circuit &ndash; Schematic created using CircuitLab
Your pull-up forms the top part of a voltage divider. When no button is pressed, the ADC will read 1023 (5V). When SW1 is pressed, the line will be pulled to ground, and the ADC will read 0 (0V). When SW2 is pressed, the voltage divider will be 47k over 2.3k, and result in a voltage of 0.23V. At 5V / 1023 = 0.0048 V per ADC step, that's roughly a result of 47. When SW3 is pressed, the voltage divider will be 47k over 2.3k + 4.3k, voltage of 0.62V, and an ADC result of 127.
As @microtherion also pointed out, use ranges. An Analog result of 0 to 30 is SW1, 31 to 100 is SW2, 101 to 150 is SW3, etc. You could use a smaller value pull-up resistor to have some wider voltage ranges. 24kΩ or 10kΩ would work, just recalculate all the voltage ranges (or just hook it up and press the buttons to read them. You could also press multiple buttons at the same time for different values, except Still/Pause, which shorts the other buttons out.