I feel I may have misspoken about Donald Trump in our Dept. Should I risk drawing attention to the issue by apologizing for it?
They say three topics are banned from polite after-dinner conversation: religion, politics and sport. There are good reasons for that.

It can be said that the same holds for workplace. Academia is, in this sense, a slightly different animal in that at times it is a place where people vigorously express their opinion, and proudly so. This does not prevent occasional ganging-ups of opinion groups against people which are not seen to conform to the currently prevalent sentiment or simple are considered an out group. 

Unless you must fear that the latter is the case (and with highly professional people, such as senior academic staff, I would start from the assumption that the threshold is relatively high for that), consider what's done is done. Let it go. 

Remember Churchill's dictum: a fanatic is someone who won't change his opinion and can't change the topic.