How do miners decide when to give up on a block?
You're right, miners can freely choose among the unconfirmed transactions when they're building their block template. However, miners are optimizing for revenue, so they'll be mostly selecting transactions in order of descending feerate.
Since transactions propagate quickly on the network, most miners will have a largely overlapping view of the queue, and their block templates should be almost the same.
Since each block commits to a specific predecessor, the best chain can only have one block at each height. Mining is progress-free in that each hash independently has a (tiny) chance of yielding a valid block. So, when a new block is found, miners do not have any sunk cost, they just update for the new predecessor, pick a new set of transactions, and go right back to mining at the new height.