What was the design decision behind that the coinbase field can have arbitrary data?
They wanted to rule out possibility that the same transaction ID would appear in different block, as a different coinbase.

There have been such in the history, but only twice, and it complicates the unspent DB application, so they just decided one day that same ID would not be distinguishable as a different transaction anymore. Thus added a protection against it, in block format version 2, so no miner can accidentally end up anymore with a non-unique ID for his coinbase transaction.