How does the reference bitcoin wallet/client handle rejected transactions? (in the API)
With Bitcoin-Qt/bitcoind there there is no callback or other real-time notification that a transaction that had previously been a valid 0/unconfirmed is not a transaction that will not confirm due to there being another confirmed transaction for the same unspent transaction output (UTXO).

When a new block is found, each transaction can be inspected (using Raw transactions) to determine the INPUT(s).   If the input was already used by a previously seen transaction, then the spend from the block would cause the earlier transaction to be an invalid double spend and it can be purged.

The Bitcoin-Qt/bitcoind client will no longer display the 0/unconfirmed transaction for a payment received that becomes invalid after a double spend from a new block arrives.