Sending an invoice per email goes to primary address, not to billing
There is no setting that you overlooked. Since Billing is not necessarily utilised it sounds like you requirement is a setting that says 'set my system to email to Billing email if it exists, otherwise send it to Primary'
(NB I realise I may not be adding much to above comments but am trying to get some unanswered questions answered as that is currently the 'weakest' aspect of our Beta civicrm stackexchange according to these stats)