How do mining pool payout methods affect the cost of executing a block withholding attack?
Generally, the higher the variance for the miners of a reward method, the higher the cost of sabotage attacks (and I think also the potential profit of lie-in-wait).

In PPS there is no variance; and the cost for sabotage is negligible, it is merely the reward of 1 share.

In a typical method with no risk absorption and moderate variance, withholding a block means you lose your portion of its reward, which is h/H, your portion of the pool hashrate.

If you take an extremely high-variance reward method such as "pay per last 10 shares", withholding blocks means losing 10% of the reward, since if you submit a block then by definition you have a share in the last 10 shares, no matter your hashrate.

And of course, when solo mining, block withholding means losing all of your mining reward.

A block finder bonus is added multiplicatively to the other costs. For example in PPL10S with 20% finder bonus, when withholding you lose both the 20% bonus and 10% of the distributed 80%, for a total cost of 28%. However, finder bonus is a poor way to combat block withholding because it creates a lot of variance.