Unexpected behavior setting non-default I2C address for HT16K33 LED Driver
The Holtek chip has a button matrix scanning feature.
When the matrix pins are not driving a LED it can read buttons when row pins are set to inputs and they have an internal pull-down. The buttons and the I2C setting strap will set input high when button is pushed or the address jumper is closed, and the correct column output is driven high to scan them. So the Toshiba driver sees this address strap as high input and drives the LEDs because it does not know why the line went high. And since the column to scan that is high, only column 0 won't have the LEDs on, and all other columns will, as your column drivers also don't know why the pins are low, whether due driving or scanning.
Also the Toshiba chip has an internal pull-down on its inputs. The value is not stated but it can easily double the load on the address selection pins. Therefore the 39k resistors may not be enough to drive them, so you might need to bring the 39k resistor to a lower value.