Can Commerce be setup to skip Cart/Customer
Yes, it's possible but would need testing out and tinkering.

You could disable the checkout pages, other than payment / offsite payment redirect

Either at 


admin/commerce/config/checkout by setting the panes to disabled
or perhaps with hook_commerce_checkout_page_info_alter and removing the pages.


You could then add a Rule 


Event: After adding product to cart 
Action: Redirect to page "checkout"


Which will hopefully take the user off to PayPal / offsite payment.

Again, needs testing, but theoretically possible.