Concurrent HTTP request loop I'm using the rolling curl library to fire HTTP requests for content-length headers of images (we need to know their size to weed out placeholders and low res images). The image URLs are stored in a database so I need to loop over the data in our products table (approx 1 million rows but will grow bigger, potentially much bigger).

I'm using PHP and the Laravel framework (the artisan CLI component). The operation seems to slow down as time progresses e.g. it starts processing 100 requests in less than a second and later the time to process 100 rows/requests is logged at over 20 seconds. Can anyone explain this and / or offer any performance improvement suggestions? The task is running on an Amazon EC2 micro instance so processing power / memory is limited.

public function fire()
{
    $dt = new DateTime();
    Log::info("started: ".$dt-&gt;format('Y-m-d H:i:s'));
    $counter = 1;
    Item2::where('img_size', '=', NULL)-&gt;chunk(1000, function($items) use ( &amp;$counter)
    {
        $results = array();
        $filePath = storage_path().'/imports/new/new_img_sizes_'.$counter.'.csv';

        if (!File::exists($filePath)) {
            File::put($filePath, '');
        }
        $start = microtime(true);
        $rollingCurl = new \RollingCurl\RollingCurl();
        $rollingCurl-&gt;setOptions($this-&gt;curlOptions);

        foreach ($items as $item)
        {
            if ($item-&gt;img !== '') {

                $results[$item-&gt;id] = array('url' =&gt; $item-&gt;img, 'size' =&gt; null);
                $rollingCurl-&gt;get($item-&gt;img);
            }

        }
        //callback runs on each curl request
        $rollingCurl-&gt;setCallback(function(\RollingCurl\Request $request, \RollingCurl\RollingCurl $rollingCurl) use (&amp;$results, $filePath) {
            $responseInfo = $request-&gt;getResponseInfo();
            //var_dump($responseInfo);exit;
            $length = $responseInfo['download_content_length'];

            foreach ($results as $key =&gt; $value) {
                if (array_search($request-&gt;getURL(),$value)) {
                    $idKey = $key;
                    $results[$idKey]['size'] = $length;
                    File::append($filePath,$idKey.','.$results[$idKey]['size']."\r\n");
                    break;
                }
            }

        })

            -&gt;setSimultaneousLimit(10)
            -&gt;execute();

        $counter++;
        echo 'done in...'.(microtime(true) - $start).PHP_EOL;
        Log::info('1000 records: '.(microtime(true) - $start));
        Log::info('Last url was: '.json_encode(end($results)));
        exit;
    }); // end item chunk


Some benchmarks:


done in...3.8803641796112
done in...7.4326379299164
done in...8.1860301494598
done in...8.5088090896606
done in...10.606615781784
done in...10.655412912369
done in...10.804574966431
done in...14.004528045654
done in...10.903785943985
done in...11.905344009399
done in...13.763195991516
done in...14.723680019379
done in...15.823812961578
done in...17.972007989883
done in...31.734715938568
done in...20.509822845459
done in...22.924754858017
done in...34.274693012238
done in...39.217702865601
done in...29.883662939072
done in...24.094554901123
done in...25.726534128189
done in...31.788655996323
done in...24.713880062103
done in...25.855134963989
done in...23.161122083664
done in...32.380167007446
done in...36.53077507019
done in...31.859884023666
done in...71.458341121674
I don't see what could be causing ever-decreasing performance since the only state outside the main loop is a counter, but adding calls to memory_get_usage as @jsanc623 recommends would help with that.

But you can definitely improve the performance by avoiding the loop over $results for every URL returned. Take advantage of PHP's array which is a hash table providing O(1) lookup.

First, make the URL the key for the results array.

foreach ($items as $item) {
    if ($item-&gt;img !== '') {
        $url = $item-&gt;img;
        $results[$url] = array(
            'id' =&gt; $item-&gt;id, 
            'url' =&gt; $url,
            'size' =&gt; null
        );
        $rollingCurl-&gt;get($url);
    }
}


Next, use isset instead of array_search and looping.

// foreach ($results as $key =&gt; $value) { ... } becomes

$url = $request-&gt;getURL();
if (isset($results[$url]) {
    $results[$url]['size'] = $length;
    File::append($filePath, $results[$url]['id'] . ',' . $length . "\r\n");
}


Since you're placing each URL into the array before making the GET request, the if isn't even necessary.

Separately, you can delay writing to the file until the end of the inner loop so you write one big chunk rather than one thousand tiny chunks. It may be that I/O is more costly, and perhaps you're getting killed by seeking to the end of an ever-growing file? Seems unlikely but maybe after a million writes it adds up.

General Review

There are a few things that could be cleaned up here as well.


Consistent formatting: put your opening brace on the same line or the next, but do it consistently.
Blank lines before the closing brace seem misplaced to me. Use blank lines to logically group related code, though functions are preferred to whitespace.
Simplify testing if the image URL exists

if ($item-&gt;img !== '')


can be simplified to

if ($item-&gt;img)


to skip null values as well. Even if the field cannot be null, it's still easier to read.