title: How to handle large amount of INSERT in Symfony2/Doctrine2 to avoid "Allowed memory size of 2147483648 bytes exhausted"
body: <p>I need to handle a <em>huge</em> amount of INSERT operations in Doctrine2 and by huge I mean around 130k or more records. This is how my code looks:</p>
  
  <pre><code>$this-&gt;_batchSize = 1000;
  
  $entTarget = $this-&gt;_em-&gt;getRepository('PDOneBundle:Target')-&gt;findAll();
  
  $currTargetArr = [];
  foreach ($entTarget as $currTarget) {
      $currTargetArr[] = $currTarget-&gt;getVeevaAccountId();
  }
  
  // $currTargetArr =&gt; is empty because there is nothing on that table
  
  echo count($finalOutput);
  
  for ($i = 0; $i &lt; count($finalOutput); $i++) {
      echo count($finalOutput[$i]['records']), "\n";
  }
  
  for ($i = 0; $i &lt; count($finalOutput); $i++) {
      for ($j = 0; $j &lt; count($finalOutput[$i]['records']); $j++) {
          $veevaAccountId = $finalOutput[$i]['records'][$j]['Id'];
  
          // if there is no territory, then we add
          if ($veevaAccountId !== null &amp;&amp; !in_array($veevaAccountId, $currTargetArr, true)) {
              // we set the values from veeva
              if (
                  $finalOutput[$i]['records'][$j]['Id'] !== null &amp;&amp;
                  $finalOutput[$i]['records'][$j]['FirstName'] !== null &amp;&amp;
                  $finalOutput[$i]['records'][$j]['LastName'] !== null
              ) {
                  $newTarget = new Entity\Target();
  
                  $newTarget-&gt;setVeevaAccountId($finalOutput[$i]['records'][$j]['Id']);
                  $newTarget-&gt;setNpi($finalOutput[$i]['records'][$j]['NPI_vod__c']);
                  $newTarget-&gt;setFirst(ucfirst(strtolower($finalOutput[$i]['records'][$j]['FirstName'])));
                  $newTarget-&gt;setLast(ucfirst(strtolower($finalOutput[$i]['records'][$j]['LastName'])));
                  $newTarget-&gt;setTitle($finalOutput[$i]['records'][$j]['Title__c']);
                  $newTarget-&gt;setDisplayName(
                      ucfirst(strtolower($finalOutput[$i]['records'][$j]['FirstName'])).' '.ucfirst(
                          strtolower($finalOutput[$i]['records'][$j]['LastName'])
                      )
                  );
  
                  $newTarget-&gt;setLastSyncAt(new \DateTime());
  
                  $this-&gt;_em-&gt;persist($newTarget);
  
                  if (($i % $this-&gt;_batchSize) === 0) {
                      echo 'Flushing batch...'."\n";
                      echo 'Memory: '.$this-&gt;getReadableSize(memory_get_usage())."\n";
  
                      $this-&gt;_em-&gt;flush();
                      $this-&gt;_em-&gt;clear(); // Detaches all objects from Doctrine!
  
                      echo 'After batch...'."\n";
                      echo 'Memory: '.$this-&gt;getReadableSize(memory_get_usage())."\n";
                  }
              }
  
              $targetArr[] = $newTarget-&gt;getId();
              $targetFailArr[] = $finalOutput[$i]['records'][$j]['FirstName'].' '.$finalOutput[$i]['records'][$j]['LastName'];
          } else {
              $entTarget = $this-&gt;_em-&gt;getRepository('PDOneBundle:Target')-&gt;find($veevaAccountId);
  
              $lastModifiedDate = new \DateTime(
                  $finalOutput[$i]['records'][$j]['LastModifiedDate']
              );
  
              if ($lastModifiedDate &gt; $entTarget-&gt;getUpdatedAt()) {
                  // obtained a territory, we update its data
                  // $entTarget-&gt;setName($finalOutput[$i]['records'][$j]['Name']);
              }
  
              $targetArr[] = $entTarget-&gt;getId();
          }
      }
  }
  
  $this-&gt;_em-&gt;flush();
  $this-&gt;_em-&gt;clear();
  </code></pre>
  
  <p>When it reach 2000 records script stop execution and fail with this message:</p>
  
  <pre><code>Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 18 bytes) in /var/www/html/reptooln_admin/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php on line 542
  
  Call Stack:
      0.0001     229280   1. {main}() /var/www/html/reptooln_admin/app/console:0
      0.0152    3246832   2. Symfony\Component\Console\Application-&gt;run() /var/www/html/reptooln_admin/app/console:27
      0.0167    3449664   3. Symfony\Bundle\FrameworkBundle\Console\Application-&gt;doRun() /var/www/html/reptooln_admin/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:126
      1.8824   54241840   4. Symfony\Component\Console\Shell-&gt;run() /var/www/html/reptooln_admin/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php:91
     14.7210   54278272   5. Symfony\Component\Console\Application-&gt;run() /var/www/html/reptooln_admin/vendor/symfony/symfony/src/Symfony/Component/Console/Shell.php:112
     14.7211   54278480   6. Symfony\Bundle\FrameworkBundle\Console\Application-&gt;doRun() /var/www/html/reptooln_admin/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:126
     14.7213   54278736   7. Symfony\Component\Console\Application-&gt;doRun() /var/www/html/reptooln_admin/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Console/Application.php:96
     14.7214   54279656   8. Symfony\Component\Console\Application-&gt;doRunCommand() /var/www/html/reptooln_admin/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:195
     14.7248   54502144   9. Symfony\Component\Console\Command\Command-&gt;run() /var/www/html/reptooln_admin/vendor/symfony/symfony/src/Symfony/Component/Console/Application.php:886
     14.7250   54507096  10. PDI\PDOneBundle\Command\PDOneSyncCommand-&gt;execute() /var/www/html/reptooln_admin/vendor/symfony/symfony/src/Symfony/Component/Console/Command/Command.php:259
     14.7251   54508672  11. PDI\PDOneBundle\Service\SyncController-&gt;syncVeevaData() /var/www/html/reptooln_admin/src/PDI/PDOneBundle/Command/PDOneSyncCommand.php:29
     15.8199   54512624  12. PDI\PDOneBundle\Service\SyncController-&gt;syncTargets() /var/www/html/reptooln_admin/src/PDI/PDOneBundle/Service/SyncController.php:95
    269.3835  719438056  13. EntityManager55881a77c7652_546a8d27f194334ee012bfe64f629947b07e4919\__CG__\Doctrine\ORM\EntityManager-&gt;flush() /var/www/html/reptooln_admin/src/PDI/PDOneBundle/Service/SyncController.php:486
    269.3835  719438104  14. Doctrine\ORM\EntityManager-&gt;flush() /var/www/html/reptooln_admin/app/cache/dev/jms_diextra/doctrine/EntityManager_55881a77c7652.php:305
    269.3835  719438104  15. Doctrine\ORM\UnitOfWork-&gt;commit() /var/www/html/reptooln_admin/vendor/doctrine/orm/lib/Doctrine/ORM/EntityManager.php:340
    269.3835  719438104  16. Doctrine\ORM\UnitOfWork-&gt;computeChangeSets() /var/www/html/reptooln_admin/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:297
    269.3835  719438104  17. Doctrine\ORM\UnitOfWork-&gt;computeScheduleInsertsChangeSets() /var/www/html/reptooln_admin/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:711
    299.1056 2146967320  18. Doctrine\ORM\UnitOfWork-&gt;computeChangeSet() /var/www/html/reptooln_admin/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:404
    299.1056 2146970736  19. ReflectionProperty-&gt;getValue() /var/www/html/reptooln_admin/vendor/doctrine/orm/lib/Doctrine/ORM/UnitOfWork.php:542
  </code></pre>
  
  <p>I'm executing this as a Symfony2 tasks so it uses CLI and not web and I call it setting 2G as max memory:</p>
  
  <pre><code>php -d memory_limit=2G app/console --shell
  </code></pre>
  
  <p>I have read several docs (<a href="http://doctrine-orm.readthedocs.org/en/latest/reference/batch-processing.html" rel="nofollow noreferrer">1</a>, <a href="http://www.doctrine-project.org/2009/08/07/doctrine2-batch-processing.html" rel="nofollow noreferrer">2</a>, <a href="https://stackoverflow.com/questions/12076177/doctrine2-batch-insert">3</a>, <a href="https://stackoverflow.com/questions/21681974/doctrine-2-bulk-insert-update-memory">4</a>, <a href="https://stackoverflow.com/questions/21931067/doctrine-2-bulk-insert-with-relation">5</a> ...) but is not helping me at all, can any give me some ideas in how to handle this? </p>
  
  <p><strong>Update: memory consumption is less but only got 2k inserts</strong></p>
  
  <p>After write some <strong>debug</strong> info to see what is happening on my logic I notice that only the first 2k items are persisted and not the rest and I can't find the cause. See this output:</p>
  
  <pre><code>php -d memory_limit=1G app/console pdone:sync --no-debug
  
  iterations
  ------------------------------------------------------------------
  0- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-2000
  1- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-4000
  2- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-6000
  3- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-8000
  4- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-10000
  5- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-12000
  6- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-14000
  7- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-16000
  8- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-18000
  9- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-20000
  10- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-22000
  11- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-24000
  12- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-26000
  13- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-28000
  14- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-30000
  15- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-32000
  16- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-34000
  17- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-36000
  18- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-38000
  19- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-40000
  20- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-42000
  21- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-44000
  22- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-46000
  23- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-48000
  24- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-50000
  25- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-52000
  26- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-54000
  27- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-56000
  28- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-58000
  29- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-60000
  30- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-62000
  31- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-64000
  32- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-66000
  33- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-68000
  34- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-70000
  35- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-72000
  36- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-74000
  37- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-76000
  38- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-78000
  39- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-80000
  40- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-82000
  41- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-84000
  42- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-86000
  43- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-88000
  44- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-90000
  45- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-92000
  46- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-94000
  47- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-96000
  48- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-98000
  49- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-100000
  50- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-102000
  51- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-104000
  52- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-106000
  53- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-108000
  54- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-110000
  55- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-112000
  56- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-114000
  57- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-116000
  58- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-118000
  59- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-120000
  60- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-122000
  61- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-124000
  62- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-126000
  63- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-128000
  64- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-130000
  65- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-132000
  66- instancePath: /services/data/v28.0/query/01g8000002eusZCAAY-134000
  
  total count:
  ------------------------------------------------------------------
  682000
  
  count per iteration:
  ------------------------------------------------------------------
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  2000
  1733
  </code></pre>
  
  <p>Why just the first 2k are persisted? What happen with the rest of items on each iteration?</p>