title: Useragent in PHP scraper script
body: <p>I have a PHP scraper script which I use to scrape a page on my site. The script then parses the content into HTML and outputs it for the user. I came across using the useragent function in PHP to pretend that you are a crawler, for example GoogleBot. How can I combine my two scripts together so the page I am scraping thinks I am a crawler?</p>
  
  <p>My scraper PHP code is:
      
  
  <pre><code>$query=$_REQUEST['q'];
  
  $html = file_get_contents("search.php?q=$query");
  preg_match_all(
      '/&lt;div class="cl1 cld"&gt;.*?&lt;a rel="nofollow" class="l le" href="(.*?)"&gt;(.*?)&lt;\/a&gt;.*?&lt;div class="cra"&gt;(.*?)&lt;\/div&gt;.*?&lt;div class="clud"&gt;(.*?)&lt;\/div&gt;.*?&lt;\/div&gt;/s',
      $html,
      $posts, // will contain the blog posts
      PREG_SET_ORDER // formats data into an array of posts
  );
  
  foreach ($posts as $post) {
      $link = $post[1];
      $title = $post[2];
      $description = $post[3];
      $url = $post[4];
  
  echo "&lt;div class='result'&gt;&lt;div class='title'&gt;&lt;a href='$link'&gt;$title&lt;/a&gt;&lt;/div&gt;$description&lt;div class='url'&gt;$url&lt;/div&gt;&lt;/div&gt;";
  }
  
  ?&gt;
  </code></pre>
  
  <p>I have this line of code which pretends to be a crawler.</p>
  
  <pre><code>$userAgent = 'MyScraperBot (http://www.mysite.com/)';
  </code></pre>