I'm looking for tutorial in web-crawlers
Question: I need to learn more about web-crawlers for a personal project of mine, and
I'd like to have some questions answered:

1) By what I've heard, it seems like Google uses python for his web-crawlers,
is that right?

2) Following that question, would you say that it is a good choice? Is Python
the most suited language for that kind of things? Why?

3) What's legal to do with web-crawlers, and what is not? I've heard that many
website won't really appreciate that you download too many of their pages, but
isn't that exactly what Google do? It seems like a big grey area and I'd like
to know how I can make sure what I do is legal...

4) If you have any good tutorial on how to make web-crawlers (the programming
language isn't important), I would really appreciate a link to it!

Thank you, and sorry for mistakes, english isn't my native language...


Answer: > 1) By what I've heard, it seems like Google uses python for his web-
> crawlers, is that right?

An early version of Google used Python for the webcrawler. This is indicated
in early publications dating back to the 90s (see [Anatomy of a Search
Engine](http://infolab.stanford.edu/~backrub/google.html).) Only a Google
employee can tell you whether they still use Python today for their crawler.

> 2) Following that question, would you say that it is a good choice? Is
> Python the most suited language for that kind of things? Why?

There is no way to answer this objectively. It's almost equivalent to asking
"is Python a good language?" Pros of using Python for a webcrawler include the
various web and networking libraries, parsers, etc. which are readily
available in Python, as well as the fact that Python's performance drawbacks
aren't likely to matter much for most webcrawlers, since a webcrawler
typically spends all of it's time waiting for URLs to resolve and download.

> 3) What's legal to do with web-crawlers, and what is not? I've heard that
> many website won't really appreciate that you download too many of their
> pages, but isn't that exactly what Google do? It seems like a big grey area
> and I'd like to know how I can make sure what I do is legal...

It's not a question of legality so much as it is a question of politeness. If
you bombard a webserver with too many requests at once, it's likely to ban
your IP. Commercial webcrawlers such as GoogleBot generally use crawl-wait
times, so that there is a delay between requests to the same host. A well-
behaved webcrawler also must obey the [Robots Exclusion
Protocol](http://en.wikipedia.org/wiki/Robots_exclusion_protocol).

> 4) If you have any good tutorial on how to make web-crawlers (the
> programming language isn't important), I would really appreciate a link to
> it!

Conceptually, a webcrawler is just a FIFO queue of URLs. You choose a starting
URL, place it at the top of the queue, pop and download the URL, parse the
HTML and extract the links, and then add each link to the queue. Then pop the
top URL and repeat ad infinitum or until some stopping condition occurs.

However, in practice, a robust webcrawler is much more complicated to write
because you need to account for many, many issues, such as avoiding
downloading duplicate URLs, implementing crawl-wait times for each host,
obeying the Robots Exclusion Protocol, avoiding [crawler
traps](http://en.wikipedia.org/wiki/Spider_trap), etc. You also probably need
to include logic for some kind of decent crawl strategy so your crawler
doesn't spend most of it's time downloading useless garbage, as the signal-to-
noise ratio of the modern World Wide Web is pretty abysmal.

