Do I need N confirmations if I'm only worried about the transfer getting invalidated by a different block?
A transaction can only be invalidated by an attempt to double-spend. That is, the same transaction can be in block X or block X+1 or block X+n as long as there is no double-spend associated with it.

If you want to make sure that a transaction goes into a block soon-ish, try to add a fee that's sufficient to make a miner want to add it to a block. In that case, you only need a couple confirmations as being an orphan block after that is exceedingly unlikely.