crawler by using urls
Question: Hi evry one i need to crawl a sit by using python(my python version is 2.6.6)
only using urls.

  * first i need to open this link(<http://www.google.com/>) and crawl,and i should collect all http links which are present in this page;
  * second, from the collected http links,i should go to each and every link, through this link i need to open and do the crawling process and should save the crawled links in a separate folder. iam just trying in this way,if any one is know the solution plz reply i need your help..


Answer: Step1: Use `urllib or urllib2` library of python and visit your page.

Step2: Use any xml/html parser like [lxml](http://lxml.de/) or [Beautiful
Soup](http://www.crummy.com/software/BeautifulSoup/) to fetch all the links
from your web page.

Step3: Now again pass these links to urllib or urllib2 and save your webpage.

And you can also give [Scrapy](http://scrapy.org/) or
[Mechanize](http://wwwsearch.sourceforge.net/mechanize/) a try.

And here is an example of using `xpath expressions with lxml`. You can get the
xpath of any element using `firebug`.

    
    
    import lxml
    from lxml import etree
    import urllib2
    
    response = urllib2.urlopen('your url')
    html = response.read()
    link = etree.HTML(html)
    links = link.xpath('xpath expr')
    

