How to exclude non-physical products from shipping calculations?
Do you have separate product types? Commerce Physical has a function that determines whether a line item is shippable: commerce_physical_line_item_shippable().

This function looks at the weight of an item. If the weight of the item is 0, then it considers it to be not-shippable.

You can also implement hook_commerce_physical_line_item_shippable_alter() and define your own logic to determine if something is shippable.

It looks like Commerce USPS implements this functionality and so you should be able to depend on this. Ideally, you'd have different product types for your physical and virtual products and your virtual products would have no weight causing them to not be factored into the shipping calculations.