Python script for web scraping from web pages to find ip address for urls present in it
Question: I have started writing script as mentioned below

    
    
    import urllib2
    from bs4 import BeautifulSoup
    
    trg_url='http://timesofindia.indiatimes.com/'
    req=urllib2.Request(trg_url)
    handle=urllib2.urlopen(req)
    page_content=handle.read()
    soup=BeautifulSoup(page_content,"html")
    new_list=soup.find_all('a')
    
    
    for link in new_list:
        print link.get('href')
    

but now i am stuck, as i am getting below mentioned output

    
    
    http://mytimes.indiatimes.com/?channel=toi
    https://www.facebook.com/TimesofIndia
    https://twitter.com/timesofindia
    https://plus.google.com/117150671992820587865?prsrc=3
    http://timesofindia.indiatimes.com/rss.cms
    https://www.youtube.com/user/TimesOfIndiaChannel
    javascript:void(0);
    http://timesofindia.indiatimes.com
    javascript://
    http://beautypageants.indiatimes.com/
    http://photogallery.indiatimes.com/
    http://timesofindia.indiatimes.com/videos/entertainment/videolist/3812908.cms
    javascript://
    /life/fashion/articlelistls/2886715.cms
    /life-style/relationship/specials/lsspeciallist/6247311.cms
    /debatelist/3133631.cms
    

please guide me to extract the different URLs present in web page and there IP
address


Answer: Use the socket module to get the ip address:

    
    
    import urllib2
    from bs4 import BeautifulSoup
    import socket
    import csv
    
    trg_url='http://timesofindia.indiatimes.com/'
    req=urllib2.Request(trg_url)
    handle=urllib2.urlopen(req)
    page_content=handle.read()
    soup=BeautifulSoup(page_content,"lxml")
    new_list=soup.find_all('a')
    
    final_list = []
    for link in new_list:
        l = link.get('href')
        try:
            final_list.append([l,socket.gethostbyname(l.split('/')[2])])
        except:
            final_list.append([l,[]])
    
    with open('output.csv','wb') as f:
        wr = csv.writer(f)
        for row in final_list:
            wr.writerow(row)
    

