Migrate with Cloud Files fopen:Too many open files I'm using migrate to import 2800 blog posts. Each have a header image which is getting imported. I'm using Rackspace Cloud Files as my CDN so the system is copying the image files up to Rackspace Cloud Files, but after it's been running for awhile I start getting this error:


  (/home/drupal/html/sites/all/libraries/rackspace/php-cloudfiles/cloudfiles.php:608)
  Invalid response: (curl error: 35) error:02001018:system
  library:fopen:Too many open files: Failed [error] to obtain valid HTTP
  response.


What is causing the problem?
I believe there is a bug in the rackspace cloudfiles api which causes connections to not be closed properly. In normal, web-based, circumstances this isn't a problem because at the end of the processing all of the connections are closed and you never usually open enough connections to have a problem.

However, I'm running the migration via drush from the command line which means the process isn't ending for a VERY long time after A LOT of connections and eventually I hit the limit for the max number of open "files."

This isn't a perfect solution, but it works even if it's hackish.

class MyArticleMigration extends BasicExampleMigration {
    public function __construct($rangeStart = 0) {
        parent::__construct();

        // code to set up the query where I pull my source data from the database.
        $query-&gt;range($rangeStart, 35);

        // rest of the code...
    }    
}


class MyArticle0Migration extends MyArticleMigration {
    public function __construct() {
        parent::__construct(0);
    }
}

class MyArticle1Migration extends MyArticleMigration {
    public function __construct() {
        parent::__construct(200);
    }
}

class MyArticle2Migration extends MyArticleMigration {
    public function __construct() {
        parent::__construct(400);
    }
}

// etc...


Then on the command line I just run:

drush migrate-import MyArticle0Migration
drush migrate-import MyArticle1Migration
drush migrate-import MyArticle2Migration
drush migrate-import MyArticle3Migration


etc...