Panel + views + taxonomy depth problem
You should try to add a "Content pane" display on the view instead of using a page display (enable "Views content panes" module). "Content pane" allows to set the argument input from the views settings. On the "argument input" you can select "From context" as source, and the required term context.

EDIT: Like I've wrote on the following comments, the correct contextual filter settings is:

1) Content: Has taxonomy term ID (with depth). Provide a default value: Taxonomy term ID from URL (Load default filter from term page). Specify validation criteria: Taxonomy term, Filter value type: Term ID

2) Content: Has taxonomy term ID depth modifier