Importing large amount of data slow process with cron and queue I am currently importing large amount of data into Drupal and I am using the hook_cron with the queue API and so far everything is working. Data is been imported as nodes and the relationship between them is created and the views are displaying the data properly.

The problem I am having is how long it is the process taking to do the import of the data. I tried using the "Ultimate Cron" to see if a parallel processing of the cron job will speed things up, but I had no luck.

I have my queue full of pending data to be process and I can think of a way to speed it up. I read about creating multiple queues, but I haven't try that.

The server is a pretty decent server and the server load is less than 2% all the time, swap is empty and I am only using 800MB of 16 GB of RAM.
I think it will be good to use multiple queues and separate php non blocking (shell_exec) process to read each queue, the idea is to call a php script to run in background which will pick a queue data and create nodes or whatever.