Get cron to process QueueWorker items at a specific time
Copied from \Drupal\Core\Annotation\QueueWorker, the annotation and placd where the @QueueWorker annotation is documented.

 * Worker plugins are used by some queues for processing the individual items
 * in the queue. In that case, the ID of the worker plugin needs to match the
 * machine name of a queue, so that you can retrieve the queue back end by
 * calling \Drupal\Core\Queue\QueueFactory::get($plugin_id).
 *
 * \Drupal\Core\Cron::processQueues() processes queues that use workers; they
 * can also be processed outside of the cron process.
 *
 * Some queues do not use worker plugins: you can create queues, add items to
 * them, claim them, etc. without using a QueueWorker plugin. However, you will
 * need to take care of processing the items in the queue in that case. You can
 * look at \Drupal\Core\Cron::processQueues() for an example of how to process
 * a queue that uses workers, and adapt it to your queue.


See the last paragraph, the solution is simply to just fill an arbitrary queue and then implement similar logic as Cron::processQueues() for your queue to process the items when you want. Nothing will happen automatically with items of a queue that doesn't have a worker plugin.

Instead of a plugin, you can implement your login in a service instead and call that from your own cron implementation.