Why are recumbent bicycles and velomobiles illegal in UCI bicycle racing?
This seems like a pretty good history of the circumstances around the UCI's banning of the recumbent. Your guess seems to be about right: recumbents were perceived as too weird. It's interesting to note that Faure had been racing on recumbents with success and official imprimatur for a while before they were banned. According to that article, there were safety concerns about fairings even back then.

The UCI has generally gotten more and more restrictive about what qualifies as a "bike" (although they have relaxed some rules after creating them—read the Wikipedia article on the hour record for some insight into the back-and-forth) and their current rationale is basically that a member of the public should be able to look at a racing bike and recognize it as a regular bike. There is certainly a chicken-and-egg aspect to this: if more people rode recumbents (or other configurations of bike), those would look like regular bikes. There is also the argument that achievements in cycling should be more attributable to the cyclist than the bike.

I can speculate about other factors playing into these decisions, but that's all it would be—speculation.