Is there a name for complement of recursively enumerable set?
The complement of a recursively enumerable (RE) language is called co-recursively enumerable (co-RE).

As RE and co-RE form the first level of the arithmetic hierarchy, they are also known by aliases $\Sigma_1^0$ and $\Pi_1^0$.