Does choosing rounding mode have to be costly?
Is this because there is something particularly complicated about rounding modes, or is it because global flags interact badly with pipelines?


The rounding mode affects how the FPU operates, and especially how it reacts to specific problems (e.g. division by -zero).

Thus, if you didn't flush the pipeline, you could getting an FPU-caused interrupt at a place where your code should assume that can't happen.


  For example, if the rounding mode were supplied as a couple of bits in the instruction, instead of as global flags, would it still be costly?


If the pipeline is smart (and complicated) enough, it'll not actually have one global state, but "shadow" states for each stage in the pipeline that is affected or affects specific global flags, so that it "looks" as if the processor wasn't pipelined w.r.t. to global flags. 

That's, in the end, equivalent to having different opcodes instead of global flags. And in that case, the pipeline doesn't have to be flushed.

However, having shadow state can be pretty costly, especially when you not only pipeline your processor, but also add speculative execution units, because that's state you'll need to duplicate for every pipeline stage. (oh, and more, harder to deal with, side channel for SPECTRE-style privilege transgressions ;) .)