Alternative approaches to "primary" record in child table
The other design option that came in my mind is to use CustomerID column in child table like Addresses and Contacts so that you will be making entry first in Customer table and that CustomerID will be referred in other table.

As suggested in comment by  @a_horse_with_no_name, you can create Unique Index as 

CREATE UNIQUE INDEX UniquePrimaryAddresss ON Addresses(CustomerID) WHERE IsPrimary = 1


Similar indexes on Contacts table as well