Migrate Module: How to construct a map on a UNION query
Either do 2 migrations. Or create a derivative table of your term data to easily set the $query for the migration before running the migration task.

You can do this by writing a custom php script to be called by drush php-script or by using the preImport migrate hook to build/rebuild the derivative table.