title: PHP cURL sitemap generator
body: <p>Hi All I am having a problem with my sitemap generator it works great on a few thousand URLS but when it gets to 20000 URL it starts to go wrong is there anything else i can be doing to help prevent this </p>
  
  <pre><code>$request_url ="http://www.example.com";//put your url here
  $url=str_replace("http://", "", $request_url); // i added this because to check if wwwdomain.com is in the url
  $alllinks=array();// create an array for all the links
           $alllinks2=array();// create an array for all the links
            $alllinks3=array();// create an array for all the links
              $newUrl=array();// create an array for all the links
  $badUrls=array("basket", "#" ,"mailto", "javascript:document", "reviews.php", "review.php","tab=",".JPG",".jpg","png","PNG","gif","GIF","item_id=","../", "pn_pr");// array of urls we dont want to include
  $ch = curl_init(); //search for links
  curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
  curl_setopt($ch, CURLOPT_URL, $request_url); 
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); 
  curl_setopt ($ch,CURLOPT_VERBOSE,false);
  curl_setopt($ch, CURLOPT_TIMEOUT, 900);
  $result = curl_exec($ch);
  </code></pre>
  
  <p>Thank you</p>