Mail.app Assumes Google is the Desired Service for a Self-Hosted Domain
I was able to finally resolve this. Here's a simplified version of the breakdown (in a corrected order), in hopes that it is helpful to anyone trapped in this in the future:


Setup the self-hosted service
Switch DNS records to point to the self-hosted service
Ensure your Mac or iOS device is aware of this DNS change (e.g. dig domain.com mx in Terminal returns what you expect to see)
Uninstall Gmail and Calendar from the Google Apps for Domains Control Panel (your data will not be lost; you can reactivate them later)
Remove the Google Apps account from your device
Add a new email account with the correct information:

After clicking "Continue" on the screen where you enter your name, email address and password, immediately press "Cancel", then
"Continue" again. This ensures no automatic search for services is
done. (Apparently in a different way than holding the option key
does...)
On each subsequent screen, hold down the option key when clicking
continue and ensure the advanced settings are correct
Confirm the account displays as "IMAP" or "POP" and not "Google
IMAP" or "Google POP"

Add any other accounts.


If you experience trouble with the above, verify that you are certain your local device sees the correct DNS information, remove all accounts from Mail and delete ~/Library/Mail.

Hopefully this helps someone else.