How much compensation does an unruly passenger have to pay to an airline if the flight has to be diverted to remove them?
Unruly passengers can face the following:


Monetary fines
Prison sentences
A ban from that airline


Excerpt from Wikipedia:


  In the United States, passengers who disrupt the duties of a flight
  crew member can face fines up to $25,000 and sometimes lengthy prison
  sentences.


IATA has a policy for these passengers mentioned here, and more information is here.

ICAO has a detailed document on unruly passenger prevention and management. They have adopted a new protocol to amend the Tokyo Convention.