Why are mining pools not removing the 250 kb block size soft cap? I might have misunderstood what is happening, but looking at the block sizes it seems that most mining pools are sticking to the 250 kb soft cap, with BTC Guild being the notable exception. There are many blocks that are just under 250 kb, so I assume that pools are excluding certain fee-paying transactions to stay below the limit. 

My question is, why are they doing so? Is the marginal cost of including those extra transactions indeed greater than the fee being paid? If not, what is their reasoning for acting in a non-profit maximizing manner?

The whole idea of Bitcoin is based on the assumption that participants behave rationally - i.e. profit maximizing. So what is going on here?
Some pools may still use the 250 KB cap for faster block propagation and reduced chance of orphaned blocks. If one block is orphaned you probably have to mine for 10 years to recover that loss with the extra transaction fees from mining larger blocks.

However I think many pools use a max size above 250 KB. The reason most of the blocks are smaller, even when there are more transactions available, is that they set a max size for how many KB of free or minimum-fee transactions they will include. There aren't enough above-minimum-fee transactions to fill up the blocks.