Strategy for organizing category pages with Views and Panels
I don't know if you figured this out already, but I'll give a quick answer.

The magic you need: arguments / contextual filters. You want to have just one Views block that takes term IDs as arguments (in a contextual filter), so it will filter on the term present in the url. Instead of a regular views block, you can also use a Views content pane (for tighter integration with Panels).

As for Panels: you need one panel page to override the taxonomy system path. When you enable Page manager (part of CTools) some page override templates are provided, just activate the taxonomy one.

Add a variant (you can optionally add a selection rule if you want this override to only work on terms of one vocabulary) and add your Views block. Since this is a taxonomy system path, term IDs are already available as arguments for the view block (Panels is also excelent for manipulating which arguments the page will provide).

Page manager, Panels and Views are complex so it's a good idea to understand how it works. I recommend checking out NodeOne's Learn Page manager screencasts (especially Views content panes), and maybe also parts of the Views series (especially contextual filters).