<p>I just started thinking about creating/customizing a web crawler today, and know very little about web crawler/robot etiquette. A majority of the writings on etiquette I've found seem old and awkward, so  I'd like to get some current (and practical) insights from the web developer community.</p>

<p>I want to use a crawler to walk over "the web" for a super simple purpose - "does the markup of site XYZ meet condition ABC?".</p>

<p>This raises a lot of questions for me, but I think the two main questions I need to get out of the way first are:</p>

<ul>
<li>It feels a little "iffy" from the get go -- is this sort of thing acceptable?</li>
<li>What specific considerations should the crawler take to not upset people?</li>
</ul>

[{'answer_id': 32381, 'author': 'Einar', 'author_id': 2964, 'author_profile': 'https://Stackoverflow.com/users/2964', 'pm_score': 2, 'selected': False, 'text': "<p>I'd say that it is very important to consider how much load you are causing. For instance, if your crawler requests every object of a single site, more or less at once, it might cause load problems for that particular site.</p>\n\n<p>In other words, make sure your crawler is not too aggressive.</p>\n"}, {'answer_id': 32392, 'author': 'Will Dean', 'author_id': 987, 'author_profile': 'https://Stackoverflow.com/users/987', 'pm_score': 4, 'selected': True, 'text': "<p>Obey robots.txt (and not too aggressive like has been said already).</p>\n\n<p>You might want to think about your user-agent string - they're a good place to be up-front about what you're doing and how you can be contacted.</p>\n"}, {'answer_id': 32452, 'author': 'Ross', 'author_id': 2025, 'author_profile': 'https://Stackoverflow.com/users/2025', 'pm_score': 2, 'selected': False, 'text': '<p>It\'s perfectly accetable to do - just make sure it only visits each page once for each session.  As you\'re technically creating a searchbot you must obey robots.txt and <code>no-cache</code> rules. People can still block your bot specifically if needed by blocking IPs.</p>\n\n<p>You\'re only looking for source code as far as I can tell so you\'ll want to build something to follow <code>&lt;link&gt;</code>s for stylesheets and <code>&lt;script src="..."&gt;&lt;/script&gt;</code> for JavaScripts.</p>\n'}, {'answer_id': 32457, 'author': 'Sean', 'author_id': 3290, 'author_profile': 'https://Stackoverflow.com/users/3290', 'pm_score': 2, 'selected': False, 'text': '<p>Load is a big consideration.  Put limits on how often you crawl a particular site and what is the most basic info you need to accomplish your goal.  If you are looking for text do not download all images, stuff like that.</p>\n\n<p>Of course obey robots.txt but also make sure your user agent string includes accurate contact info and maybe a link to a web page describing what you are doing and how you do it.  If a web admin is seeing a lot of requests from you and is curious you might be able to answer a lot of questions with an informative web page.</p>\n'}, {'answer_id': 32491, 'author': 'Ricardo Reyes', 'author_id': 3399, 'author_profile': 'https://Stackoverflow.com/users/3399', 'pm_score': 2, 'selected': False, 'text': '<p>Besides WillDean\'s and Einar\'s good answers, I would really recommend you take a time to read about the meaning of the HTTP response codes, and what your crawler should do when encountering each one, since it will make a big a difference on your performance, and on wether or not you are banned from some sites. </p>\n\n<p>Some useful links:</p>\n\n<p><a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html" rel="nofollow noreferrer">HTTP/1.1: Status Code Definitions</a></p>\n\n<p><a href="http://diveintomark.org/tests/client/http/" rel="nofollow noreferrer">Aggregator client HTTP tests</a></p>\n\n<p><a href="http://en.wikipedia.org/wiki/List_of_HTTP_status_codes" rel="nofollow noreferrer">Wikipedia</a></p>\n'}, {'answer_id': 32634, 'author': 'ceejayoz', 'author_id': 1902010, 'author_profile': 'https://Stackoverflow.com/users/1902010', 'pm_score': 2, 'selected': False, 'text': '<p>Please be sure to include a URL in your user-agent string that explains who/what/why your robot is crawling.</p>\n'}, {'answer_id': 43457, 'author': 'Boaz', 'author_id': 2892, 'author_profile': 'https://Stackoverflow.com/users/2892', 'pm_score': 2, 'selected': False, 'text': '<p>Also do not forget to obey the bot meta tags: <a href="http://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.2" rel="nofollow noreferrer">http://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.2</a></p>\n\n<p>Another thing to think about - when spider pages, don\'t be too hasty deciding things don\'t exist or have errors. Some pages are offline due to maintenance work or errors that are corrected within a short period.</p>\n'}, {'answer_id': 105367, 'author': 'user9569', 'author_id': 9569, 'author_profile': 'https://Stackoverflow.com/users/9569', 'pm_score': 2, 'selected': False, 'text': '<p>All good points, the ones made here.  You will also have to deal with dynamically-generated Java and JavaScript links, parameters and session IDs, escaping single and double quotes, failed attempts at relative links (using ../../ to go past the root directory), case sensitivity, frames, redirects, cookies....</p>\n\n<p>I could go on for days, and kinda have. I have a <a href="http://searchtools.com/robots/robot-checklist.html" rel="nofollow noreferrer" title="Robots Checklist">Robots Checklist</a> that covers most of this, and I\'m happy answer what I can.</p>\n\n<p>You should also think about using open-source robot crawler code, because it gives you a huge leg up on all these issues.  I have a page on that as well:  <a href="http://searchtools.com/robots/robot-code.html" rel="nofollow noreferrer" title="Source Code for Web Robot Spiders">open source robot code</a>.  Hope that helps!</p>\n'}, {'answer_id': 105409, 'author': 'MarkR', 'author_id': 13724, 'author_profile': 'https://Stackoverflow.com/users/13724', 'pm_score': 2, 'selected': False, 'text': '<p>You will need to add some capability to blacklist sites / domains or other things (IP ranges, ASN, etc) to avoid your spider getting bogged down with spam sites.</p>\n\n<p>You\'ll need to have a HTTP implementation with a lot of control over timeout and behaviour. Expect a lot of sites to send back invalid responses, huge responses, rubbish headers, or just leave the connection open indefinitely with no response etc.</p>\n\n<p>Also don\'t trust a 200 status to mean "the page exists". Quite a large proportion of sites send back 200 for "Not found" or other errors, in my experience (Along with a large HTML document).</p>\n'}]
['https://Stackoverflow.com/questions/32366', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/326/']