How do I remove extra divs added by panels?
There are a couple ways you can tackle this. 

One option is to simply create your own panel layouts.  I often do this because I like control over the markup and I want to reduce the divitus in my theme.  It also lets me create layouts that more directly match what I need.  It's pretty easy to create your own panel layouts in your theme.  Read more here.

Another option is to override the panel template files in your theme.  Similar to any module that provides a template file, you can copy that template file to your theme and change the markup.  You'll find the panel template files in [path_to_modules]/panels/plugins/layouts/[each_layout].