Performance and optimisation when using panels with views and content pane
The primary thing you should do, is cache as much rendered output as you can. If you cache the generated markup itself, whether you're Views configuration is optimal mostly becomes a non-issue.

Views itself, can be cached with Views content cache, or if you complicated invalidation criteria, with a long timed cache and Cache actions.

More experimental, is a module I've written myself, called Panels cache, which moves the caching one layer higher up, for even better performance, but it requires that the content being cached all comes from the same entity. (I use the module in production, but would still love testing and feedback on the module)

On the actual Views configuration, you should avoid having a single with multiple panes, if you have a lot of overrides. If your displays are significantly different, you waste processing time as builds must build and then alter the display. This however, will be a very small optimization, compared to the caching solutions suggested above.

You can read about more different performance improvements in my answer on How do you improve Drupal performance?