Using Panels and Views to override taxonomy listing - which argument to use?
In the Views UI, each Content Pane has "argument settings". In there you have, at least, two options.


The easiest to configure, is "From panel argument", and its option in this case, "Second argument". This works fine in your situation.
The most flexible, that I always use, require more understanding of the Ctools context.

The argument setting then is "From context" and in this case "Taxonomy Term" -> "Name".
Save the View, go into the Panels UI, add the View Pane, not just a View.
Assuming your Panel Page now has a taxonomy term, which is always the case on the /taxonomy/term page, you will get a dropdown, most of the time with just one option, that lists the available terms that you pass on to the View.



If you for example had a path like /color/red/blue, because you wanted two terms, the second version will allow the re-use of the same View on the same page, with different content.