How to slow down migrate? Cloud Files can't keep up I'm using migrate to import 2800 blog posts. Each have a header image which is getting imported. I'm using Rackspace Cloud Files as my CDN so the system is copying the image files up to Rackspace Cloud Files, but after it's been running for awhile I start getting this error:


  (/home/drupal/html/sites/all/libraries/rackspace/php-cloudfiles/cloudfiles.php:608)
  Invalid response: (curl error: 35) error:02001018:system
  library:fopen:Too many open files: Failed  [error] to obtain valid
  HTTP response.


My -guess- is that the migrate is working too quickly and the file transfers are happening in parallel or something and I'm hitting a limit on the number of open files for Rackspace. I'm not exactly sure.

I'd like to try adding a delay in between each node. There are only 2800 and I can let it run for a little while, so I don't mind the delay as long as it'll let me complete the migrate.
In your migration scripts you can use either prepare() or complete() (http://drupal.org/node/1132582) and add in sleep(). It's a hacky solution but it should give you enough time between processing each row for Rackspace to catch up.

public function prepare($node, stdClass $row) {
  sleep(30); //wait for 30 seconds
}