How does a channel know to drive in I2S TDM mode?
However, I don't understand how the audio device knows when to drive the serial data line with it's data. How is contention avoided?

You need to design the device so that it skips as many slots as necessary, while someone else drives the serial clock. There's no bidirectional communication in I2S. There's no contention, only pre-defined slotting.
However, it feels like the usual design choice here would be that there's a single device delivering all the data, and receiving samples from 8 sources (e.g. by clocking ADCs, or by receiving from 4 stereo I2S links) in itself. That's a classical FPGA use case, but I'm sure there's also dedicated ASICs for such purposes.