Python - getting source code with socket
Question: I wanna send http get request and receive source code from webpage, this has
to be done through sockets. I set buffer size to 4096, but my script download
only small part from the page

    
    
    import socket
    sock = socket.socket ( socket.AF_INET, socket.SOCK_STREAM )
    sock.connect ( ( "edition.cnn.com", 80 ) )
    
    host = socket.gethostbyname("edition.cnn.com")
    sock.sendall('GET http://edition.cnn.com/index.html HTTP/1.1\r\n'\
        + 'User-Agent: agent123\r\n'\
        + 'Host: '+host+'\r\n'\
        + '\r\n')
    
    print sock.recv(4096)
    sock.close()
    

After I run this code data I get are

    
    
    HTTP/1.1 200 OK
    
    Server: nginx
    
    Date: Wed, 01 Jan 2014 18:31:25 GMT
    
    Content-Type: text/html
    
    Transfer-Encoding: chunked
    
    Connection: keep-alive
    
    Set-Cookie: CG=GR:44:Réthimnon; path=/
    
    Last-Modified: Wed, 01 Jan 2014 18:31:22 GMT
    
    Vary: Accept-Encoding
    
    Cache-Control: max-age=60, private
    
    Expires: Wed, 01 Jan 2014 18:32:25 GMT
    
    
    
    ac2a
    
    
    <!DOCTYPE HTML>
    <html lang="en-US">
    <head>
    <title>CNN.com International - Breaking, World, Business, Sports, Entertainment and Video News</title>
    <meta http-equiv="X-UA-Compatible" content="IE=edge"/>
    <meta http-equiv="content-type" content="text/html;charset=utf-8"/>
    <meta http-equiv="last-modified" content="2014-01-01T18:28:34Z"/>
    <meta http-equiv="refresh" content="1800;url=http://edition.cnn.com/?refresh=1"/>
    <meta name="robots" content="index,follow"/>
    <meta name="googlebot" content="noarchive"/>
    <meta name="description" content="CNN.com International delivers breaking news from across the globe and information on the latest top stories, business, sports and entertainment headlines. Follow the news as it happens through: special reports, videos, audio, photo galleries plus interactive maps and timelines."/>
    <meta name="keywords" content="CNN, CNN news, CNN International, CNN International news, CNN Edition, Edition news, news, news online, breaking news, U.S. news, world news, global news, weather, business, CNN Money, sports, politics, law, technology, entertainment, education,
    

Which isn't even first 13 rows from source code... view-
source:<http://edition.cnn.com/index.html>

* * *

And another problem, when I try address google.com like a host

    
    
    import socket
    sock = socket.socket ( socket.AF_INET, socket.SOCK_STREAM )
    sock.connect ( ( "google.com", 80 ) )
    
    host = socket.gethostbyname("google.com")
    sock.sendall('GET http://google.com/index.html HTTP/1.1\r\n'\
        + 'User-Agent: agent123\r\n'\
        + 'Host: '+host+'\r\n'\
        + '\r\n')
    print sock.recv(4096)
    sock.close()
    

I get this response

    
    
    HTTP/1.1 301 Moved Permanently
    
    Location: http://www.google.com/index.html
    
    Content-Type: text/html; charset=UTF-8
    
    Date: Wed, 01 Jan 2014 18:38:57 GMT
    
    Expires: Fri, 31 Jan 2014 18:38:57 GMT
    
    Cache-Control: public, max-age=2592000
    
    Server: gws
    
    Content-Length: 229
    
    X-XSS-Protection: 1; mode=block
    
    X-Frame-Options: SAMEORIGIN
    
    Alternate-Protocol: 80:quic
    
    
    
    <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
    <TITLE>301 Moved</TITLE></HEAD><BODY>
    <H1>301 Moved</H1>
    The document has moved
    <A HREF="http://www.google.com/index.html">here</A>.
    
    </BODY></HTML>
    

which says that page is moved to the same address like i wanted download...


Answer: `sock.recv(4096)` will read **up to** 4096 bytes; it depends on how much data
has already arrived how much can actually be returned by the call. There is no
guarantee that 4096 bytes will actually be available for reading in one go.

You'll have to _continue_ to read from the socket until all data is received:

    
    
    data = ''
    chunk = sock.recv(4096)
    while chunk:
        data += chunk
        if len(data) >= 4096:
            break
        chunk = sock.recv(4096)
    

Your request to `http://google.com/index.html` redirects to `www.google.com`,
a **different** hostname. Adjust your request accordingly.

If you wanted to implement a full-on HTTP client, you'd have to parse the
status line, process the `301` redirect response by parsing out the
`Location:` header, and making a new connection to request the new URL given
to you.

