Would bypass capacitors of a voltage regulator which powers an I2C device and the bus cause issues?
No, the capacitors are on the power supply pins, and the capacitors will have no effect on the I2C bus pins. That's exactly how it is supposed to be done.