How is it decided what modulation is used during an 802.11 communication?
As usual, it depends.

Let's restrict ourselves to the OFDM-based IEEE802.11 standards (a,g,n,ac,p, and many more). 

These have a typical frame structure, in which one symbol, which is always BPSK-modulated, contains the rate setting info (and thus, implicitly, the modulation used).

As an example: Got this graphic from keysight



See the violet symbol.

Notice that in OFDM, you can use different modulations for different subcarriers, or in fact, you can freely pick your modulation for each frame and subcarrier. You don't typically do that in Wifi (the standards restrict your freedom here, to varying degrees, depending, again, on the individual 802.11 standard); in 4G/LTE OFDM systems, you can expect a frame of the downlink signal to be aimed at a whole set of different receivers, which all pick out "their" part in the subcarrier/symbol plane – and thus, you can expect different robustnesses to apply to different ressource blocks. Yay, complexity!

In essence: for wifi-style OFDM 802.11, the transmitter decides on what modulation it uses for the payload data itself. The optimal choice depends on a lot of factors, some of which are receiver capabilities, throughput and power consumption limits, and an estimate for the interference and noise levels at the receiver.

Thus, the algorithms picking an optimal modulation for any specific instant in any specific use case for any specific x in 802.11x are very diverse, and, often, very highly optimized by the chipset/driver vendor, and thus can be very obscure.