LED and Button Matrix multiplexing?
Scanning the buttons by setting row as output high and column to input
  with pull-down.


The LEDs would still be forward biased and could be recognized as if a button was being pressed. It's borderline that this will work effectively.

When driving the LEDs, keep the row as a driver and the column as an input (with a pull down) but, when scanning the switches, reverse bias the LEDs by driving the column high and having a 10k to 100k pull down on the rows (now set as inputs): -

You could even put a diode in series with the switches to stop the LED being "blanked" when the switch is pressed. Clearly when scanning the switches the LED gets turned off for a small length of time but this should be unnoticeable if scanning switches / driving LEDs at a fast rate.



It should work at logic levels as low as 3.3 volts but check the input threshold high limits in the data sheet are not greater than about 2.6 volts.