Import large number of entries every day - 1 million I have a shopping product feed that is generated every day with around a million products. This can be in any format needed - CSV, JSON, XML etc, and will live on an S3 bucket.
Each day, I need to import this feed to entries in Craft and assign them categories etc. A large percentage of the products will be duplicates but it may need to update data within the product.
What would be the best way to import this large of a data set?
I have used FeedMe in the past, but I'm not sure whether it could handle this many products.
Thanks in advance.
You could write a service method that reads the import file and divides it into chunks, then it could dispatch a queue job for each chunk which does the actual entry creation/update. This way you could scale the resources and amount of queue workers available to work through the queue and even Play with the chunk size.
In the queue job you might want to implement proper exception handling to keep track of unsuccessful entry operations to handle them somehow.