What was the design decision behind that the coinbase field can have arbitrary data? From what I learned, the coinbase field in the block reward transaction can carry any data (As long as smaller than 100b).

Why that actually? Was it easier to implement in the beginning? I would just have defined that the coinbase transaction may have no input. I know it is useful now to carry more information (see BIP34). I'm just not sure what was Satoshi's intention here, although he included himself the famous


  The Times 03/Jan/2009 Chancellor on brink of second bailout for banks


Further it seems that this is missing in the transaction db and is unspendable, but I may confuse things here.
They wanted to rule out possibility that the same transaction ID would appear in different block, as a different coinbase.

There have been such in the history, but only twice, and it complicates the unspent DB application, so they just decided one day that same ID would not be distinguishable as a different transaction anymore. Thus added a protection against it, in block format version 2, so no miner can accidentally end up anymore with a non-unique ID for his coinbase transaction.