Are these preferences consistent with rationality?
You implicitly assume that the utility of $n$ units of $Y$ equals $n$ times the utility of 1 unit of $Y$, and there is no reason for that. For instance, if $Y$ is a fridge, the gain in utility from having 1 fridge compared to 0 is certainly larger than the gain in utility from having 2 fridges compared to 1. The formula $U("nY")=n*U("Y")$ that you use does not make sense: there are utility functions other than $U$ that represent the same preferences over $\{X,Y,Z\}$, and you will reach different conclusions if you make the same computation with these utilities.

The Von Neumann-Morgenstern theory does not make any assumption on how to value $n$ units of a good. A Von Neumann-Morgenstern representation attributes a utility level to all the consumption goods in the choice set. In your case, the representation would include objects such as $U("Y")$ (the utility of one unit of $Y$), $U("nY")$ (the utility of $n$ units of Y), $U("Z \text{ and  }(n-1)X")$ (the utility of $1$ unit of $Z$ and $n-1$ units of $X$), etc. There is nothing that allows you to relate these values with each other. The only behavioral restrictions of the Von Neumann-Morgenstern theory concern preferences over lotteries induced by these goods.

You need other tools to understand whether preferences over a finite set of commodities are rational. These tools are the Weak Axiom of Revealed Preferences and the Generalized Axiom of Revealed Preferences.

On a historical note, the reason why Von Neumann and Morgenstern developed a theory of preferences over lotteries is precisely because they wanted to identify a notion of "how much the good $Y$ is preferred to the good $X$". An obvious possibility was to observe preferences over pairs of goods such as: $n$ units of $Y$ versus $m$ units of $X$. But this is problematic for the reason stated above: nothing relates the utility of one unit of $Y$ to the utility of $n$ units of $Y$. Their trick was to consider preferences over lotteries that deliver $Y$ with a certain probability and $X$ with the complementary probability: the answer to the question "how much are you willing to pay to increase your chance of receiving $Y$ by 1%?" gives a good measure of "how much you prefer $Y$ to $X$". In contrast, the answer to the question "how many units of $X$ are you willing to sacrifice to receive one additional unit of $Y$?" does not make any meaningful sense.