How to programmatically set custom display settings for an RSS feed?
@JeremyFrench pointed out that the Features Module correctly exports the field settings for the view-mode, which fulfilled the middle step. No extra API calls is necessary.

My problem was with step 1. Though field settings are defined per-content-type and per-view-mode, the actual definition of view modes is independent of content-types. All content types share the same set of view modes, though each can implement their own field settings for that view mode.

This means that in my call to hook_entity_info_alter(), I should not have referenced customtype, I should just reference 'node'. That simple change enables everything to work.