Panel Pages and Taxonomy Term Arguments
There may be better routes out there, but I am going to suggest the one I have in mind right now. 

You could create your own custom block - that wraps around the Views block display that you already created. Why? Because that way you have full control of the arguments.

Wrapping a Views block display with your own custom block:


Implement hook_block_info() and hook_block_view().
Inside your hook_block_view() call the View display programatically. The View display should be a block (it looks like you got it already configured).
While instantiating the View display, you can grab the arguments from the URL using the arg() function. Since this is your own code, you can do anything that you want with those arguments. For example, you could load a term using an nid-based argument, get the term's proper name, URL-decode it if needed, and then feed it to your display.
Execute the View display, retrieve the results, and return them as the output of your block.


Then instead of embedding in the panel the block that was provided by your View, you embed the one that you created in your custom module. Again, since you called the View programatically, you had full control of the arguments that were passed to it.

It sounds complex, but it's really easier than you think. Here's a plain example of how to call the View (you could modify it for your needs):

$view = views_get_view('my_foobar_view');
$result = $view-&gt;execute_display('block_display_baz');
$term_id = arg(2); // Get the term from my-panel/foo/&lt;term id&gt;
$view-&gt;set_arguments(array($term_id));
$view-&gt;execute();
if(!empty($view-&gt;result)) {
  foreach($view-&gt;result as $result) {
    // Do something with $result...
  }
}


This G.D.O. document can provide you with more examples on how to programatically embed Views.