How can I tell if a page allows bots?
<p>I am trying to create a bot that checks to see if a particular URL has some particular content. However, I keep getting an 'HTTP redirection loop' error when I run it.</p>

<p>The only thing I can suspect is that the page does not allow bots. Is there any way to tell if the page does not allow bots? I have googled it, but I have yet to find an answer.</p>

<p><strong>EDIT</strong></p>

<p>After checking somethings out, this is what the robots.txt says:</p>

<pre><code>User-agent: *
Disallow: /advsched/
</code></pre>

<p>I also noticed that when I disable cookies in my browser and visit the page, I get the 'HTTP redirection loop' error. So from what I understand, the page I am trying to access is does not allow bots. However, from what I understand about cURL functions, as long as my user-agent is something like this:</p>

<pre><code>Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 GTB5
</code></pre>

<p>The site cannot tell if I am a bot or not. That leaves only one thing - cookies. I know cURL functions can process cookies, but can they handle them so that I look like a standard user? I have not been able to get it to work yet. </p>
