A huge number of buttons on a Raspberry Pi
It could certainly work, I2C can run pretty fast. Just makes sure the ICs you're using support an address space wide enough to sit on the same bus. The nice thing about I2C is that it's multimaster, if you find an IC that can let your main controller know when a button has been pushed that should work fine. Even if you have to resort to polling the raspberry pi should be able to handle that load just fine.

As for response times, you can calculate that roughly by going through timing diagrams in the datasheet, and with a little experimentation. Start with how long it takes you to acquire the data in the first place. Unless there's an awful bottleneck somewhere it should run fast enough for your intended function.