Software Mixing of Multiple Audio Streams From Network
Generally you will need some sort time sync protocol.
Typically you want create locally one circular buffer for each network channel. As soon as a packet comes in, you shove it into circular buffer ort that channel at the write positions.
Playback is clocked by your local D/A clock. As soon as the D/A buffer is running empty, you grab a frame from all the client circular buffers at the current read position, mix them together and put them into the D/A buffer.
There are two things you need to manage.

Drop out: That happens if you need a new playback frame and the circular buffer of a client is empty OR a packet got dropped (they should be numbered to detect this). There isn't a whole lot you can do about. You can play zeros or try to extrapolate from the previous buffer A drop out means that your network can't keep up.
Clock drift. Even if everyone is playing at nominally 44.1kHz, no two clocks are identical and over time they will drift apart. To fix these you need to a) measure the clock drift and b) correct it. a) can be done through either a global time reference (something like NTP), by averaging over packet arrival times, or monitoring buffer levels. b) will require some sort of sample rate conversion: the most crude option would be to occasionally drop a frame insert a zero frame.

This is age old trade off: the lower the latency, the higher your drop out rate will be. So dialing this in &quot;optimally&quot; is the key here. It's easy enough to adjust: simply make your circular buffers smaller/bigger. However finding the optimum is hard since it depends very much on current network conditions so in practice this really should be adaptive.