Prove that $D =\{x \in \mathbb{N} | \Phi_x(x)\uparrow\}$ is **not** recursively enumerable
As Andrej states, what you've (correctly) done is show that $D=\{x:\Phi_x(x)\uparrow\}$ is not recursive. However, there are plenty of sets which are recursively enumerable but not recursive, so you're not done yet: you need to show that $D$ is not the domain of any partial recursive function.

REMARK: Some texts define a set to be r.e. if it is the range of a total recursive function, or is empty. This is slightly less elegant since it requires a separate clause to handle $\emptyset$; it also generalizes less well if one looks into higher computability theory. However, regardless of what definition your book gives, the equivalence between them should be a very early exercise.

There are a couple ways to do this:


You could argue directly: suppose $D$ is the domain of $\Phi_i$. Can you use the recursion theorem to cook up a $j$ such that $\Phi_j(j)\downarrow\iff \Phi_i(j)\downarrow$? Do you see why this gives a contradiction?
Alternately, note that $D$ is the complement of the Halting Problem $H=\{x: \Phi_x(x)\downarrow\}$. Two early results in computability theory are that (i) $H$ is r.e. but not recursive and (ii) if a set and its complement are r.e., then they are each recursive. Do you see how to prove each of these facts, and apply them to the current problem? (Note that this approach isn't actually much different than the first one, since the recursion theorem is used in proving (i).)


REMARK: Some texts define the Halting Problem differently, some common alternate candidates being $\{x: \Phi_x(0)\downarrow\}$ and $\{\langle x, y\rangle:\Phi_x(y)\downarrow\}$. It's a good exercise to show that these are all r.e. and Turing equivalent (indeed $1$-equivalent!) to each other.

By similar arguments you can show that minor variations of $D$ are also not r.e. - e.g. can you show that $\{x: \Phi_x(17)\uparrow\}$ is not r.e.?



Responding to the comments: one may be tempted to try to enumerate $D$, or things like $D$, by looking at approximations given by stages. E.g. we could let $D_s=\{x:\Phi_x(x)[s]\uparrow\}$; this is the set of indices of machines which on their own input don't halt in fewer than $s$ steps. The $D_s$s are r.e. - indeed, uniformly recursive - and approximate $D$ "from above" in the sense that $D=\bigcap_{s\in\mathbb{N}} D_s$. However, this ultimately isn't relevant to this problem since the intersection of infinitely many r.e. sets need not be r.e.