How to cancel added product to cart if previous order has same commerce-product creator id
tl;dr - Rules is really the place to handle this. You need to hook into the add to cart form's validate/submit handlers and take care of it in there.

The best way to handle this is to look into the answer provided here: How do I prevent a product from being added to the cart?

This answer was originally written to do that very thing: to prevent a product from being added to an order if some criteria were met (and the context was a marketplace multi-cart as well).