Who does the Warlord pay to -- the bank or the player who had their buildings destroyed?
I have two reasons to conclude you pay to the bank:


The rulebook doesn't say that you have to pay to the bank when building a district. This is assumed because it wouldn't make sense otherwise. Similarly, if the rulebook doesn't say whom to pay to destroy, then I assume it must go to the bank.
The warlord can destroy one of it's own district if he wants to replace it. The warlord still has to pay the amount, which then obviously would not go to himself but to the bank.