How do you determine which baud rate to choose, its relation with MCU clock and oversampling at RX
1) There's only two points, how much data you must send within some given time frame, and which baud rates your microcontroller is capable of with the master clock they have. Some have only simple baud rate generators that divide with an integer, some have more complex ones that have a fractional baud rate generator. For example, an AVR running at 4 MHz crystal can't go up to 38400 baud unless you switch the oversampling from 16x to 8x with U2X bit. Even then, the baud rate will be 0.16% off from 38400 baud, but the error is insignificant.

2) You can't send more bits per second than what your bits per second rate is! It defines the rate of bits you can send. If you need to send more, then increase baud rate.

3) Synchronous serial does use a bit clock for the data. But it does not mean that a 16 MHz CPU uses 16 MHz clock for everything. You can still use a divided down baud rate clock for example for 9600 baud rate, just as you would with only an internal clock except it stays within the chip and does not come out.

4) Baud rates have to match, but only to a certain degree that is within tolerance limits. By oversampling you have only one sixteenth of time difference between the sampled start bit edge and the real received edge, so you can start counting when to sample the middle of the data bits. So the clocks only need to be sufficiently close to each other during transmission of one byte, and depending on a lot of factors, it does not need to be more than 1% to 2% percent accurate. Which means devices with less precise clocks can be used and make them cheaper.

4.1) That is how it is in general done. Sample the middle, but to avoid a spike of noise just at the sampling moment, majority logic takes in 3 samples and gives out the result.

4.2) Of course you could change oversampling from 16 to only 8, but then you need tighter tolerance baud rate clocks, because you can be almost up to one eighth of a bit already wrong when detecting the start bit edge, instead of only one sixteenth. The sampling of three bits is for combating a noise spike at the sampling instance, if there would only be one sampling instance.