Can Panels just be uninstalled if you want to switch themes?
Usually, uninstalling removes everything created by the module, its setting and its database tables.

But for what you are after, you don't even need to uninstall them, not even disabling to be honest, just don't use panels tools in your new theme.

Considering that, I would delay the idea of removing them and the old theme to after the new one is ready.
Just because of "how i did it in the other theme?" kind of questions