Pressing one key in a button matrix reads all keys in that column
The columns need a pull down, I see no evidence of that above.
The way you have the diodes facing the buttons will pull the column upwards, so you need to counter that with a pull-down (so you can see a low signal when the button is inactive).
Alternatively you could swap the signal directions and polarities and send a low signal on the columns and receive it on the rows. (with pull-up on the rows)