How does peer banning work for onion peers?
There's no handling of this really. The incoming connection just appears to be from localhost, bitcoind has absolutely no way of even identifying what circuit they came from. Banning them becomes a nothing or everything affair. It would be extremely unwise to act in the way that you have defined, especially given that at this time there are less than 50 connectable peers, and there's no guarantee they aren't all run by the same operator. 

Realistically Bitcoin has very little defense against any meaningful denial of service attacks. It is very vulnerable to volumetric, protocol level (especially prior to the removal of BIP37), and resource exhaustion attacks by even a relatively unfunded attacker, regardless of the network being used as a transport.