How do I properly pass taxonomy term from panels to the context filter in views?
it depends how you are trying to get the taxonomy term, but here is one solution. In my example I'm using a node as my context, and a taxonomy field of that node as the contextual argument.

There are basically two things you need to get this to work. One, you need to use view panes, so you can get the arguments from the panels context. 

How can I show a view in panel with argument in drupal 7

Second, if your trying to get the taxonomy term from a field you need to convert the term to a term id using validation.

Views contextual filters taxonomy name

For instance this would work if you were trying to create a panel that diplayed a node, and a view of content related by a certain taxonomy term that that node belongs to.