Should I use both caches pane and panels?
If you panel already in cache, it doesn't matter that you panes are in cache, because cache of panel are alredy include panes.

But if you panel cache have lifetime around 1 hour and some panes have lifetime around 1 day (but not all) it can be usefull and take more performance to you.

So, cache of panels and panes for the same time doesn't take more effect then cache for panels only.