Trouble opening page with Python requests
Question: I am having some trouble opening a page with Python requests package. The page
opens fine in a browser, but the program will just hang when attempting to get
the site.

    
    
    import bs4, requests
    
    link = "http://s6.mediastreaming.it:8080/"
    r = requests.get(link)
    data=r.text
    
    soup = bs4.BeautifulSoup(data)
    soup.prettify()
    print soup
    

Any help understanding why this site hangs would be greatly appreciated. The
code works fine with <http://google.com> as the link.

Edit: Added info to help. This is a site that is streaming music. I just want
to scrape the part of the page that says what the current song is. That is
all. But maybe the fact that the page is a music streaming site is causing the
issue? I just want the text of the source. Nothing else.

Edit 2: Tried the following to see if I was indeed getting a stream rather
than the page I wanted.

    
    
    import bs4, requests
    
    link = "http://s6.mediastreaming.it:8080/"
    r = requests.get(link,stream=True)
    
    filename = "testfile.txt"
    
    with open(filename,'wb') as fd:
        for chunk in r.iter_content(100):
            fd.write(chunk)
    

Here's what I got as the output:

    
    
    ICY 200 OK
    icy-notice1:<BR>This stream requires <a href="http://www.winamp.com/">Winamp</a><BR>
    icy-notice2:SHOUTcast Distributed Network Audio Server/Linux v1.9.8<BR>
    icy-name:Pig Radio - The Best Electronic & Indie Pop/Rock 24/7
    icy-genre:Eclectic
    icy-url:http://www.pigradio.com
    content-type:audio/mpeg
    icy-pub:1
    icy-br:128
    
    &Oç)goiYQŠ < 6Ê !‡À¡ö³ ‡/OFÌ)8…¨ÐU!ðiÁP¡¢ãÅ..........
    

When I open the page in a browser and view source I get the following:

    
    
    <HTML><HEAD><meta http-equiv="Content-Language" content="en-us"><meta http-equiv="Content-Type" content="text/html; charset=windows-1252"><meta http-equiv="Pragma" content="no-cache"><meta http-equiv="Expires" content="Mon, 01 Jan 1990 12:00:00 GMT"><title>SHOUTcast Administrator</title><style type="text/css"><!--a:link {color: blue; font-family:Arial, Helvetica; font-size:9pt;}a:visited {color: blue; font-family:Arial, Helvetica; font-size:9pt;}a:hover {color: red; font-family:Arial, Helvetica; font-size:9pt; }.default {color: White; font-family:Arial, Helvetica; font-size:9pt; font-weight: normal}.ST {color: White; font-family:Arial, Helvetica; font-size:8pt; font-weight: normal}.logoText {color: red; font-family: Arial Black, Helvetica, sans-serif; font-size: 25pt; font-weight: normal; letter-spacing : -2.5px;}.flagText {color: blue; font-family: webdings; font-size: 36pt; font-weight: normal; }.ltv {color: blue; font-family: Arial, Helvetica, sans-serif; font-size: 9pt; font-weight: normal;}.tnl {color: black; font-family: Arial, Helvetica, sans-serif; font-size: 10pt; font-weight: bold; text-decoration: none;}--></style></HEAD><BODY topmargin=0 leftmargin=0 marginheight=0 marginwidth=0 bgcolor=#000000 text=#EEEEEE link=#001155 vlink=#001155 alink=#FF0000><font class=default><table width=100% border=0 cellpadding=0 cellspacing=0><tr><td height=50><font class=flagText>U</font><font class=logoText>&nbsp;SHOUTcast D.N.A.S. Status</font></td></tr><tr><td height=14 align=right><font class=ltv><a id=ltv href="http://www.shoutcast.com/">SHOUTcast Server Version 1.9.8/Linux</a></font></td></tr><tr><td bgcolor=#DDDDDD height=20 align=center><table width=100% border=0 cellpadding=0 cellspacing=0><tr><td align=center><font class=tnl><a id=tnl href="index.html">Status</a></font></td><td align=center><font class=tnl>&nbsp;|&nbsp;</font></td><td align=center><font class=tnl><a id=tnl href="played.html">Song History</a></font></td><td align=center><font class=tnl>&nbsp;|&nbsp;</font></td><td align=center><font class=tnl><a id=tnl href="listen.pls">Listen</a></font></td><td align=center><font class=tnl>&nbsp;|&nbsp;</font></td><td align=center><font class=tnl><a id=tnl href="home.html">Stream URL</a></font></td><td align=center><font class=tnl>&nbsp;|&nbsp;</font></td><td align=center><font class=tnl><a id=tnl href="admin.cgi">Admin Login</a></font></td></tr></table></td></tr></table><br><table cellpadding=5 cellspacing=0 border=0 width=100%><tr><td bgcolor=#000025 colspan=2 align=center><font class=ST>Current Stream Information</font></td></tr></table><table cellpadding=2 cellspacing=0 border=0 align=center><tr><td width=100 nowrap><font class=default>Server Status: </font></td><td><font class=default><b>Server is currently up and public.</b></td></tr><tr><td width=100 nowrap><font class=default>Stream Status: </font></td><td><font class=default><b>Stream is up at 128 kbps with <B>45 of 300 listeners (45 unique)</b></b></td></tr><tr><td width=100 nowrap><font class=default>Listener Peak: </font></td><td><font class=default><b>200</b></td></tr><tr><td width=100 nowrap><font class=default>Average Listen Time: </font></td><td><font class=default><b>7h&nbsp;13m&nbsp;12s</b></td></tr><tr><td width=100 nowrap><font class=default>Stream Title: </font></td><td><font class=default><b>Pig Radio - The Best Electronic & Indie Pop/Rock 24/7</b></td></tr><tr><td width=100 nowrap><font class=default>Content Type: </font></td><td><font class=default><b>audio/mpeg</b></td></tr><tr><td width=100 nowrap><font class=default>Stream Genre: </font></td><td><font class=default><b>Eclectic</b></td></tr><tr><td width=100 nowrap><font class=default>Stream URL: </font></td><td><font class=default><b><a href="http://www.pigradio.com">http://www.pigradio.com</a></b></td></tr><tr><td width=100 nowrap><font class=default>Stream AIM: </font></td><td><font class=default><b><a href="aim:goim?screenname=N/A">N/A</a></b></td></tr><tr><td width=100 nowrap><font class=default>Stream IRC: </font></td><td><font class=default><b><a href="http://www.shoutcast.com/chat.phtml?dc=N%2FA">N/A</a></b></td></tr><tr><td width=100 nowrap><font class=default>**Current Song: </font></td><td><font class=default><b>Midlake - Young Bride (Cassettes Won't Listen Remix)**</b></td></tr></table><br><table cellpadding=0 cellspacing=0 border=0 width=100%> <tr><td bgcolor=#DDDDDD  nowrap colspan=5 align=center><table cellspacing=0 cellpadding=0 border=0><tr><td><font class=ltv>Written by Stephen 'Tag Loomis, Tom Pepper and Justin Frankel</font></td></tr></table></td></tr><tr><td nowrap colspan=5 align=center><font class=ST><b><a href="http://www.shoutcast.com/disclaimer.phtml">Copyright Nullsoft Inc</a><a href="/llamacookie">.</a> 1998-2004</b></font></td></tr></table></font></body></html>
    

Obviously it's this last bit of HTML that I want to scrape for the "Current
Song" How do I get just this HTML??

Edit3: Solved it! I used Wireshark to capture the GET that the browser was
sending and added all the parameters I saw there to the header of my Python
GET. It looks like this:

    
    
    import bs4, requests, urllib2
    
    link = "http://173.255.137.244:8080/"
    
    filename = "testfile.txt"
    
    payload = {'Host':'173.255.137.244:8080','User-Agent':'Mozilla/5.0 (Windows NT 6.1; rv:26.0) Gecko/20100101 Firefox/26.0','Accept-Encoding':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8','Accept-Language':'en-US,en;q=0.5','Accept-Encoding':'gzip, deflate'}
    r = requests.get(link,headers=payload)
    data = r.text
    
    soup = bs4.BeautifulSoup(data)
    soup.prettify()
    print soup
    

Moral of the story. Wireshark is cool.


Answer: It seems like the site has some approaches preventing spiders from crawling.

If there's no authentication needed to crawl a page and the spider has
something wrong when crawling, then there're usually two ways to do:

  * Add http request headers when emitting a request which makes it like visiting the page by a web browser. Some sites usually check the `User-Agent` field to see if it's a spider. 
  * Add some cookies needed when emitting a request. Some sites need some information from cookies when responding. 

Apart from wireshark which is a little complex, observing the network requests
and its headers by the chrome developer tools in chrome is a good choice.

