Useragent in PHP scraper script
<p>I have a PHP scraper script which I use to scrape a page on my site. The script then parses the content into HTML and outputs it for the user. I came across using the useragent function in PHP to pretend that you are a crawler, for example GoogleBot. How can I combine my two scripts together so the page I am scraping thinks I am a crawler?</p>

<p>My scraper PHP code is:
    

<pre><code>$query=$_REQUEST['q'];

$html = file_get_contents("search.php?q=$query");
preg_match_all(
    '/&lt;div class="cl1 cld"&gt;.*?&lt;a rel="nofollow" class="l le" href="(.*?)"&gt;(.*?)&lt;\/a&gt;.*?&lt;div class="cra"&gt;(.*?)&lt;\/div&gt;.*?&lt;div class="clud"&gt;(.*?)&lt;\/div&gt;.*?&lt;\/div&gt;/s',
    $html,
    $posts, // will contain the blog posts
    PREG_SET_ORDER // formats data into an array of posts
);

foreach ($posts as $post) {
    $link = $post[1];
    $title = $post[2];
    $description = $post[3];
    $url = $post[4];

echo "&lt;div class='result'&gt;&lt;div class='title'&gt;&lt;a href='$link'&gt;$title&lt;/a&gt;&lt;/div&gt;$description&lt;div class='url'&gt;$url&lt;/div&gt;&lt;/div&gt;";
}

?&gt;
</code></pre>

<p>I have this line of code which pretends to be a crawler.</p>

<pre><code>$userAgent = 'MyScraperBot (http://www.mysite.com/)';
</code></pre>
