Improve performance by using drush functions instead of Drupal module functions
Queues are defined within Drupal modules, and you can use drush to run those queues:

drush @site queue-list
drush @site queue-run queue_name


Changing the interface to the import code (queue, batch api or a drush command) won't affect drastically the import time.

I'd review the import code to try and optimize it. For example by caching (in memory) any data you may be loading recurringly from the database.