How do mining pool payout methods affect the cost of executing a block withholding attack? I'm wondering how each payout method (PPS, DGM, PPLNS, Proportional, etc) influences the cost of executing a block withholding attack.

Does it matter how the payout method places variance between pool and miners? With DGM this is a setting that can be adjusted by the pool operator. With other payout methods it is inherent to the system.

Does a block finder's bonus influence the cost of an attack? By that I mean part of the block income being reserved for the miner who created it, before the rest is distributed according to the pool's chosen payout method.

Are other factors important?

Which systems would be expensive to attack, and which would be cheapest?
Generally, the higher the variance for the miners of a reward method, the higher the cost of sabotage attacks (and I think also the potential profit of lie-in-wait).

In PPS there is no variance; and the cost for sabotage is negligible, it is merely the reward of 1 share.

In a typical method with no risk absorption and moderate variance, withholding a block means you lose your portion of its reward, which is h/H, your portion of the pool hashrate.

If you take an extremely high-variance reward method such as "pay per last 10 shares", withholding blocks means losing 10% of the reward, since if you submit a block then by definition you have a share in the last 10 shares, no matter your hashrate.

And of course, when solo mining, block withholding means losing all of your mining reward.

A block finder bonus is added multiplicatively to the other costs. For example in PPL10S with 20% finder bonus, when withholding you lose both the 20% bonus and 10% of the distributed 80%, for a total cost of 28%. However, finder bonus is a poor way to combat block withholding because it creates a lot of variance.