Max speed of communication between the switched device of I2C switch IC
The datasheet states clearly that the maximum clock frequency is 400kHz. I can find no indication that there would be a different clock rate for communications to the device itself.

So, no, you can not operate at a clock frequency of 1MHz and expect reliable operation.