Wires not making proper contact/interfering
This is what's called bouncing.  You should expect any mechanical switch to do that for up to a few 10s of ms on make and break.

To deal with this, you perform de-bouncing in software.  Usually that means you don't consider a new button state valid until you've seen that state continually for 50 ms or so.

If the bouncing persists while you're holding the button down, then the button is faulty.  Replace it.