Importing large datasets via HTTP I'm having trouble importing a very large dataset with feeds. It's about 8mb of text in XML format. It imports impossibly slow (I started it about a month ago and it's still not finished).

However, if I run the exact same importer with the same file I upload from my computer, it runs in about two minutes.

The thing is, I need to check the stock of my products from my supplier through this file every hour. Most of the data is irrelevant other than stock info.

Are there any good alternatives to do what I'm trying to do? (Drupal Commerce)
I've been thinking that if I have another module retrieve the file from my supplier every hour, then I could just point feeds to the file on my own server. However, I'm having trouble finding a module that does that file retrieve.

Any ideas?
A month-long operation is a good sign that something went wrong? How did you launch the import? If you used a Web GUI, the operation probably timed-out without updating the UI. I doubt your webserver is configured to allow a month-long HTTP request processing. Are you able to connect to the server to check if a PHP process is indeed still running?

Instead of looking for a module to download the imported file every hour, you could rely on a simple cron job using curl or wget to download the file:

0 * * * * wget http://my.supplier.com/products.xml -a /var/log/supplier-products-download.log --no-verbose -O /some/path/products.xml


If I remember correctly, the default XML parser for Feeds uses SimpleXML. It will load the complete XML tree in memory beofore processing it. For a 8Mb file this could start to be an issue (depending on your PHP memory configuration). A beast approach would be to write your own Feed parser (as a Feeds plugin) using the XMLReader API which support a streaming mode (ie. data are processed when reading the file).