Detecting headers status takes too long I have a table in my database with 35000 unique URLs. I use them to create my XML sitemaps for the site. I have setup crontab jobs to create automatically new URLs, delete non existing URLs and determine if a URL is valid.

For the last part I use the following function:

function get_url_status($url)
{
    $httpCode = 0;
    $headers = get_headers($url);
    $http_header_info = $headers[0];
    $httpCode = substr($http_header_info, 9, 3);

    if ($httpCode=='200')
    {
        // check in page for noindex and dissable that url
        $metas = get_meta_tags($url);
        if ( isset($metas['robots']) &amp;&amp; strpos( strtolower($metas['robots']),  'noindex')!==false  )
        {
            $httpCode = 410;
        }
    }

return $httpCode;
}


This will return to me the HTTP status code - 200 for legal URLs that will be included in sitemap, 400 for not found, 410 for URLs with noindex in them, 301 for redirections, etc.

In my XML sitemaps I include only URLs with status code 200.

My problem is that this function takes around 40 minutes to check the URLs in database. Is there a way to speed things up?
So you're fetching 70k pages.  You could half that if you only fetch the page once and then do what get_headers and get_meta_tags do manually.  That will presumably take now 20 minutes.

Then you can try doing this in parallel, so run two instances of the script with one half of the URLs each.