Is there a general name for the setup in which payoffs are not known exactly but players try to influence each other's perception of the payoffs?
What players are trying to do is always up for interpretation, it is not coded into the mathematics of game theory. 

The madman strategy can be modelled as a Bayesian game, with different types having different payoffs, and one player sending a signal about their type, the other player observing the signal. A situation in which types are indistinguishable based on their signals is called a pooling equilibrium.