What is the theoretical result of flattening a list containing only itself?
From a mathematical point of view, this is impossible: there is no such a thing as a set containing itself (in normal logic sense) or a vector who has itself as an element, so this question doesn't really mean anything outside the realm of using pointers in a list
BUT, on a second thought: I might be able to formulate what &quot;flattening&quot; is if you allow me to define things however I want.
Lets say our world is some graph $G$, nodes in it would be elements, and an edge $(v,u)$ would mean that the element $v$ contains the pointer to $u$. Then, in those terms, the array [[5,6],3,[7]] might look like this:

Flattening the graph in this sense means to take every non-direct child of the root node (the array's node) and make it a direct child of it.
Then, your array X looks like this:

And when you flatten it you would remain with the exact same graph, and therefore in this sense when you flatten X you would get simply X itself.
However, this is truly meaningless because it's only by the definition I just made up: you could have defined it in another way and it would have resulted in something else maybe!