Comparing Responses for Linear and Non-Linear System in Simulink
Ah HA!.  You used a Matlab nonlinear function block, but you're misunderstanding the system equation.  The function \$\dot{\vec x} = f(\vec x, u)\$ is coughing up the derivative of \$\vec x\$, not \$\vec x\$ itself.  You need to have a function block that just finds \$\dot{\vec x}\$ from \$\vec x\$ and \$u\$, then feeds it to an integrator (Simulink should be able to integrate a vector just fine) and feeds the \$\vec x\$ back to the block, and extracts \$y\$ from it.

Here's a block diagram of what you want to achieve (sorry for the crappy picture).  I can't remember my Simulink well enough to say exactly how to do it -- I suspect you can make a multi-input, multi-output block, but if you can't you can make a concatinated input vector with x and u, and concatenate y and x-dot in the output, and use muxing and demuxing to get the data in and out.