title: Reading using urllib2.urlopen()
body: <p>I am trying to use urllib2 module in python to fetch content of a url.</p>
  
  <p>Let's say my url is "<a href="http://chortle.ccsu.edu/AssemblyTutorial/Chapter-01/ass01_12.html" rel="nofollow">http://chortle.ccsu.edu/AssemblyTutorial/Chapter-01/ass01_12.html</a>"</p>
  
  <p>When I try to fetch it's content using these two simple line it gives me the complete html content of it.</p>
  
  <pre><code>response = urllib2.urlopen(url)
  content = response.read()
  print(content)
  </code></pre>
  
  <p>However when I redefine this thing inside a function it return me an html without the content of the body tag in it.</p>
  
  <pre><code>def getContentURL(url):
      ''' returns the html content of the given url '''
      response = urllib2.urlopen(url)
      content = response.read()
      return content
  
  content = getContentURL(url)
  soup = BeautifulSoup(conten) #added in edit
  print(content)
  </code></pre>
  
  <p>I only get this much.</p>
  
  <pre><code>&lt;!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"     "http://www.w3.org/TR/html4/loose.dtd"&gt;
  &lt;html&gt;
  &lt;head&gt;
   &lt;meta content="text/html;charset=utf-8" http-equiv="Content-Type"/&gt;
   &lt;meta content="Bradley Kjell kjell at ieee dot org " name="author"/&gt;
   &lt;meta content="2007" name="copyright"/&gt;
   &lt;meta content="index,follow" name="robots"/&gt;
   &lt;title&gt;
    CHAPTER 1 — Introduction
   &lt;/title&gt;
   &lt;link href="../AssemblyStyle.css" rel="stylesheet" type="text/css"/&gt;
  &lt;/head&gt;
  &lt;body&gt;
  &lt;/body&gt;
  &lt;/html&gt;
  </code></pre>
  
  <p>Why is this happening ? I am unable to understand this strange behavior.</p>
  
  <p>=============================== Edit ===============================================
  So I wrote a test.py with the same thing and it works perfectly.</p>
  
  <pre><code>import os
  from bs4 import BeautifulSoup
  import urllib2
  import urllib
  
  def getContentURL(url):
      ''' returns the content of the given url in text format '''
      response = urllib2.urlopen(url)
      content = response.read()
      return content
  
  url = "http://chortle.ccsu.edu/AssemblyTutorial/Chapter-01/ass01_1.html"
  
  content = getContentURL(url)
  soup = BeautifulSoup(content)
  print(content) #prints everything
  print(soup) #prints without the body's inner html
  
  for link in soup.find_all('a'):
      #print(link)
      print(link.get('href'))
  </code></pre>
  
  <p>However same lines of code in my original code doesn't works, which has some other stuffs in the beginning. The link to it is <a href="https://github.com/kumar116/WebsiteCopier/blob/master/web_save.py" rel="nofollow">https://github.com/kumar116/WebsiteCopier/blob/master/web_save.py</a>. Posting a link as it's big to paste here.</p>
  
  <p>The only change of line that you will see there is I am printing 
  print(soup.prettify()) or print(soup).</p>
  
  <p>Which eats up everything inside my body tag.</p>
  
  <p>I need the soup in order to be able to parse the html.</p>