Documentation or Tutorial on using Batch or Queue Services API Programmatically I am working on importing content into my drupal 8 website from an external API that returns XML data. The calls to the API take a long time to process. When I make too many queries at once it takes that much longer and it causes my site to timeout. 

I have been trying to find online some direction on how to use the Batch API Service or the Queue Service programmatically in D8 but haven't as of yet found any good tutorials or documentation on this. Anyone have any good suggestions on where I can look or direction on where to start to use batch or queue services in D8? I have found many tutorials for D7 but haven't been successful in getting those two work with my D8 code.

Any suggestions would be greatly appreciated!
I have made an example module to help understanding the Queue API and the Batch API. It also shows how to use them together.

I show you here the files and its contents. I have made a lots of comment, so i think, it is easy to understand.

xml_import_example.info.yml

type: module
name: XML import example
package: Examples
description: "This module helps understanding the Batch API and Queue API with an XML import example"
core: 8.x


xml_import_example.permissions.yml

import content from xml:
  title: 'Import content from xml'
  description: 'With this permission user can import contents from a XML source'
  restrict access: TRUE


xml_import_example.routing.yml

# Get contents from the xml source
xml_import_example.get_contents_from_xml:
  path: '/get-contents-from-xml'
  defaults: { _controller: '\Drupal\xml_import_example\Controller\ImportContentFromXML::getContentsFromXMLPage' }
  requirements:
    _permission: 'import content from xml'
# Process all queue items with batch
xml_import_example.process_all_queue_items_with_batch:
  path: '/process-all-queue-items'
  defaults: { _controller: '\Drupal\xml_import_example\Controller\ImportContentFromXML::processAllQueueItemsWithBatch' }
  requirements:
    _permission: 'import content from xml'


src/Controller/ImportContentFromXML.php

&lt;?php
/**
 * @file
 * Contains \Drupal\xml_import_example\Controller\ImportContentFromXML.
 */

namespace Drupal\xml_import_example\Controller;

use Symfony\Component\DependencyInjection\ContainerInterface;
use Drupal\Core\Controller\ControllerBase;
use Drupal\Core\Queue\QueueWorkerManager;
use Drupal\Core\Queue\QueueFactory;

/**
 * You can use this constant to set how many queued items
 * you want to be processed in one batch operation 
 */
define("IMPORT_XML_BATCH_SIZE", 1);

class ImportContentFromXML extends ControllerBase {

  /**
   * We add QueueFactory and QueueWorkerManager services with the Dependency Injection solution
   */

  /**
   * @var QueueFactory
   */
  protected $queueFactory;

  /**
   * @var QueueWorkerManager
   */
  protected $queueManager;

  /**
   * {@inheritdoc}
   */
  public function __construct(QueueFactory $queue_factory, QueueWorkerManager $queue_manager) {
    $this-&gt;queue_factory = $queue_factory;
    $this-&gt;queue_manager = $queue_manager;
  }

  /**
   * {@inheritdoc}
   */
  public static function create(ContainerInterface $container) {
    $queue_factory = $container-&gt;get('queue');
    $queue_manager = $container-&gt;get('plugin.manager.queue_worker');

    return new static($queue_factory, $queue_manager);
  }

  /**
   * Get XML from the API and convert it to 
   */
  protected function getContentsFromXML() {
    // Here you should get the XML content and convert it to an array of content arrays for example
    // I use now an example array of contents:
    $contents = array();

    for ($i = 1; $i &lt;= 20; $i++) {
      $contents[] = array(
        'title' =&gt; 'Test title ' . $i,
        'body' =&gt; 'Test body ' . $i,
      );
    }

    // Return with the contents    
    return $contents;
  }

  /**
   * Page where the xml source is preprocessed
   */
  public function getContentsFromXMLPage() {
    // Get contents array
    $contents = $this-&gt;getContentsFromXML();

    foreach ($contents as $content) {
      // Get the queue implementation for import_content_from_xml queue
      $queue = $this-&gt;queue_factory-&gt;get('import_content_from_xml');

      // Create new queue item
      $item = new \stdClass();
      $item-&gt;data = $content;
      $queue-&gt;createItem($item);
    }

    return array(
      '#type' =&gt; 'markup',
      '#markup' =&gt; $this-&gt;t('@count queue items are created.', array('@count' =&gt; count($contents))),
    );
  }

  /**
   * Process all queue items with batch
   */
  public function processAllQueueItemsWithBatch() {

    // Create batch which collects all the specified queue items and process them one after another
    $batch = array(
      'title' =&gt; $this-&gt;t("Process all XML Import queues with batch"),
      'operations' =&gt; array(),
      'finished' =&gt; 'Drupal\xml_import_example\Controller\ImportContentFromXML::batchFinished',
    );

    // Get the queue implementation for import_content_from_xml queue
    $queue_factory = \Drupal::service('queue');
    $queue = $queue_factory-&gt;get('import_content_from_xml');

    // Count number of the items in this queue, and create enough batch operations
    for($i = 0; $i &lt; ceil($queue-&gt;numberOfItems() / IMPORT_XML_BATCH_SIZE); $i++) {
      // Create batch operations
      $batch['operations'][] = array('Drupal\xml_import_example\Controller\ImportContentFromXML::batchProcess', array());
    }

    // Adds the batch sets
    batch_set($batch);
    // Process the batch and after redirect to the frontpage
    return batch_process('&lt;front&gt;');
  }

  /**
   * Common batch processing callback for all operations.
   */
  public static function batchProcess(&amp;$context) {

    // We can't use here the Dependency Injection solution
    // so we load the necessary services in the other way
    $queue_factory = \Drupal::service('queue');
    $queue_manager = \Drupal::service('plugin.manager.queue_worker');

    // Get the queue implementation for import_content_from_xml queue
    $queue = $queue_factory-&gt;get('import_content_from_xml');
    // Get the queue worker
    $queue_worker = $queue_manager-&gt;createInstance('import_content_from_xml');

    // Get the number of items
    $number_of_queue = ($queue-&gt;numberOfItems() &lt; IMPORT_XML_BATCH_SIZE) ? $queue-&gt;numberOfItems() : IMPORT_XML_BATCH_SIZE;

    // Repeat $number_of_queue times
    for ($i = 0; $i &lt; $number_of_queue; $i++) {
      // Get a queued item
      if ($item = $queue-&gt;claimItem()) {
        try {
          // Process it
          $queue_worker-&gt;processItem($item-&gt;data);
          // If everything was correct, delete the processed item from the queue
          $queue-&gt;deleteItem($item);
        }
        catch (SuspendQueueException $e) {
          // If there was an Exception trown because of an error
          // Releases the item that the worker could not process.
          // Another worker can come and process it
          $queue-&gt;releaseItem($item);
          break;
        }
      }
    }
  }

  /**
   * Batch finished callback.
   */
  public static function batchFinished($success, $results, $operations) {
    if ($success) {
     drupal_set_message(t("The contents are successfully imported from the XML source."));
    }
    else {
      $error_operation = reset($operations);
      drupal_set_message(t('An error occurred while processing @operation with arguments : @args', array('@operation' =&gt; $error_operation[0], '@args' =&gt; print_r($error_operation[0], TRUE))));
    }
  }
}


src/Plugin/QueueWorker/ImportContentFromXMLQueueBase.php

&lt;?php

/**
 * @file
 * Contains Drupal\xml_import_example\Plugin\QueueWorker\ImportContentFromXMLQueueBase
 */

namespace Drupal\xml_import_example\Plugin\QueueWorker;

use Drupal\Core\Plugin\ContainerFactoryPluginInterface;
use Drupal\Core\Queue\QueueWorkerBase;
use Drupal\Core\Queue\SuspendQueueException;
use Symfony\Component\DependencyInjection\ContainerInterface;
use Drupal\node\Entity\Node;

/**
 * Provides base functionality for the Import Content From XML Queue Workers.
 */
abstract class ImportContentFromXMLQueueBase extends QueueWorkerBase implements ContainerFactoryPluginInterface {

  // Here we don't use the Dependency Injection, 
  // but the create method and __construct method are necessary to implement

  /**
   * {@inheritdoc}
   */
  public function __construct() {}

  /**
   * {@inheritdoc}
   */
  public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition) {
    return new static();
  }

  /**
   * {@inheritdoc}
   */
  public function processItem($item) {
    // Get the content array
    $content = $item-&gt;data;
    // Create node from the array
    $this-&gt;createContent($content);
  }

  /**
   * Create content
   *
   * @return int
   */
  protected function createContent($content) {
    // Create node object from the $content array
    $node = Node::create(array(
      'type'  =&gt; 'page',
      'title' =&gt; $content['title'],
      'body'  =&gt; array(
        'value'  =&gt; $content['body'],
        'format' =&gt; 'basic_html',
      ),
    ));
    $node-&gt;save();
  }
}


src/Plugin/QueueWorker/ImportContentFromXMLQueue.php

&lt;?php

namespace Drupal\xml_import_example\Plugin\QueueWorker;

/**
 * Create node object from the imported XML content
 *
 * @QueueWorker(
 *   id = "import_content_from_xml",
 *   title = @Translation("Import Content From XML"),
 *   cron = {"time" = 60}
 * )
 */
class ImportContentFromXMLQueue extends ImportContentFromXMLQueueBase {}


So this is the working module, you can test it in you site.

If you visit the /get-contents-from-xml URL 20 queue items are made from a contents array.

The src/Plugin/QueueWorker/ImportContentFromXMLQueue.php contains this annotation:
cron = {"time" = 60}

So if you run cron, the queue items are processed for maximum 60 seconds.
You can increase or decrease this time, with that annotation.

If you remove the cron = {"time" = 60} line, cron do nothing with your queue items.

If you would like to process all the queue items in you browser, you have to visit the following url:
/process-all-queue-items

It will collect all of your queue items, creates batch operations from them, and after that it process one after another.

I hope you and others with the same question understand the usage of the Queue API and Batch API with the help of this module.

Have a nice day!