Relation between SER and the channel capacity
Yes, there is a relationship between SER and channel capacity.  The channel capacity equation is-

$$
C = B\log_2(1 + \frac{S}{N})
$$
where $C$ is the channel capacity in bits/s, $B$ is the bandwidth in Hz, $S$ is the average received signal power, and $N$ is the average noise power across the bandwidth $B$.

Looking at the equation, it is clear that increasing the SNR will increase the channel capacity.  Increasing SNR does, of course, also decrease the symbol error rate (SER).  Thus, there is an inverse relationship between SER and channel capacity.


  What is {Modulation Order x (1-SER) / symbol duration [bps]} called?


As far as I am aware it does not have a name.  Having a bit stream where some of the bits are good and some of the bits are bad is generally not very useful.  Shannon's theorem assumes that error correction codes are used to reduce the number of errors to an arbitrarily low rate.  This reduces the number of generated bits/s, but increases the reliability of those bits.  The "channel capacity" of a channel is the number of bits it can produce with an arbitrarily low error rate.