How do economists model VNM-rationality violation?
It appears you are looking for literature on Ambiguity Aversion and/or "Uncertainty Aversion".
You can start by looking up the work of L.G Epstein, I. Gilboa, and  D. Schmeidler.

It is an attempt to formalize the behavior of agents exhibiting behavior consistent with the Ellsberg paradox.