Insert task to Queue only if unique
Create a class to extend the systemqueue. I called mine Unique. This will serialise the data use db_merge to insert only if it doesn't exist.

class Unique extends SystemQueue {

  public function createItem($data) {

    // save doing this twice
    $serial_data = serialize($data);

    // ignoring expire
    $query = db_merge('queue')
    -&gt;key(array('name' =&gt; $this-&gt;name, 'data' =&gt; $serial_data))
    -&gt;fields(array(
        'name' =&gt; $this-&gt;name,
        'data' =&gt; $serial_data,
        'created' =&gt; time(),
    ));
    return (bool) $query-&gt;execute();
  }
}


You then add the variables to hook_install() or hook_enable().

variable_set('queue_class_[your queue name]', 'Unique');


matching [your queue name] to the queue name, e.g if you used this:

DrupalQueue::get('update_profile_tasks')-&gt;createQueue();


the variable would be:

variable_set('queue_class_update_profile_tasks', 'Unique');


Thanks to this for getting me on track. Updated answer to be cleaner thanks to comment suggestions.