Is there a term for simulated irrationality?
I don't think there's an official term for this kind of bluffing specifically, that's widely used. But Nixon's approach to the Soviet Union, to act in an exaggerated irrational manner to deter them from provoking the US, is sometimes referred to as the "Madman Strategy".