Why using Hyper-threading can lead to performance degradation
It would seem that the answer, as per usual, has something to do with caches. Since hyper-threads utilize the same L1 and L2 caches, one hyper-thread can trash another's caches. In the worst case the threads take turns trashing the other's cache and performance degrades as the caches are re-filled only to be trashed again.

Of course, this is likely very workload specific, which would explain why the problem is mostly seen on servers.

It is also worth noting that the answer depends in part on what you mean by "performance." For example, in real-time systems your performance metric is predictability instead of throughput, which hyper-threading negatively impacts by introducing unpredictable stalls.