Commerce Address Book – need to reuse customer profile without address
It turns out that the Commerce Addressbook Extra module takes care of this. In case anyone else is searching after a smooth solution.
Link to module