title: Getting Through Proxy Server with Python
body: <p>I am trying to get through a proxy server through python to extract some information from the website, so far I this piece of code, but it doesnt seem to be working</p>
  
  <pre><code>import requests
  import BeautifulSoup
  
  
  
  URL = 'http://proxy.library.upenn.edu/login?url=http://clients1.ibisworld.com/'
  
  session = requests.session()
  
  
      # This is the form data that the page sends when logging in
  login_data = {
      'pennkey': "****",
      'password': "****",
      'submit': 'login',
  }
  
      # Authenticate
  r = session.post(URL, data=login_data)
  
  
  doc = BeautifulSoup.BeautifulSoup(r.content)
  
  
  print doc
  </code></pre>
  
  <p>edit: this is what this prints: </p>
  
  <pre><code>Gorkems-MacBook-Pro:desktop gorkemyurtseven$ python extract.py
  &lt;!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"&gt;
  &lt;html&gt;
  &lt;head&gt;
  &lt;meta name="HandheldFriendly" content="True" /&gt;
  &lt;meta name="viewport" content="width=device-width, height=device-height, user-scalable=yes, minimum-scale=.5" /&gt;
  &lt;title&gt;University of Pennsylvania Libraries Proxy Service - Login&lt;/title&gt;
  &lt;link href="/public/proxysm.css" media="print, screen" rel="stylesheet" type="text/css" /&gt;
  &lt;script language="javascript"&gt;
      function validate(){
          var isgoldcard = document.authenticate.pass.value;
          var isgoldcardRegxp = /00000/;     
          if (isgoldcardRegxp.test(isgoldcard) == true)
          alert("Authentication is by PennKey only.");  
      }
  &lt;/script&gt;
  &lt;script type="text/javascript"&gt;
    var _gaq = _gaq || [];
    _gaq.push(['_setAccount', 'UA-982196-4']);
    _gaq.push(['_trackPageview']);
  
    (function() {
      var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
      ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
      var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
    })();
  
  &lt;/script&gt;
  &lt;!--[if IE]&amp;gt;
  &amp;lt;style&amp;gt;
  table, form .limitwidth {width: 252px;}
  .holdsubmit {width: 143px;}
  &amp;lt;/style&amp;gt;
  &amp;lt;![endif]--&gt;
  &lt;/head&gt;
  &lt;body onload="document.authenticate.user.focus();"&gt;
  &lt;div id="logostripe"&gt;
  &lt;div&gt;&lt;a href="http://www.library.upenn.edu/"&gt;&lt;img src="/public/librarieslogologin.gif" border="0" alt="Penn Libraries Home" /&gt;&lt;/a&gt;&lt;/div&gt;
  &lt;/div&gt;
  &lt;h1&gt;Libraries Proxy Service&lt;/h1&gt;
  &lt;div id="holder"&gt;
  &lt;form name="authenticate" action="https://proxy.library.upenn.edu/login" method="post" autocomplete="off"&gt;
  &lt;div class="limitwidth"&gt;
  &lt;input type="hidden" name="url" value="http://clients1.ibisworld.com/" /&gt;
  &lt;script type="text/javascript"&gt;
              var t = location.search;
              t = t.substr(t.indexOf('proxySessionID')+15,t.indexOf('&amp;amp;')-16);
              document.cookie="proxySessionID="+escape(t)+"; path=/; domain=.library.upenn.edu";
          &lt;/script&gt;
  &lt;table align="center" cellspacing="0" cellpadding="2" border="0"&gt;
  &lt;tr&gt;
  &lt;td class="holdlabels"&gt;&lt;label for="user"&gt;PennKey:&lt;/label&gt;&lt;/td&gt;
  &lt;td&gt;&lt;input type="text" name="user" /&gt;&lt;/td&gt;
  &lt;/tr&gt;
  &lt;tr&gt;
  &lt;td&gt;&lt;label for="password"&gt;Password:&lt;/label&gt;&lt;/td&gt;
  &lt;td&gt;&lt;input type="password" name="pass" onblur="validate(); return false;" /&gt;&lt;/td&gt;
  &lt;/tr&gt;
  &lt;tr&gt;
  &lt;td&gt;&lt;/td&gt;
  &lt;td class="holdsubmit"&gt;
  &lt;div&gt;&lt;input type="submit" value="Login" /&gt;&lt;/div&gt;
  &lt;/td&gt;
  &lt;/tr&gt;
  &lt;/table&gt;
  &lt;/div&gt;
  &lt;/form&gt;
  &lt;ul class="moreinfo"&gt;
  &lt;li&gt;&lt;a class="menuitem" href="http://www.upenn.edu/computing/pennkey"&gt;PennKey information&lt;/a&gt;&lt;/li&gt;
  &lt;/ul&gt;
  &lt;div class="notices"&gt;
    The Library Proxy Service allows you to use
  domain-restricted resources &amp;amp; services by authenticating yourself as Penn Faculty,
  Student, or Staff. 
  &lt;/div&gt;
  &lt;div class="alert"&gt;
  
  Please note limitations on the use of restricted online resources. 
  &lt;br /&gt;&lt;br /&gt;
  PennKey holders must be current faculty, student, or staff, have valid University PennCommunity credentials and abide by stated &lt;a href="http://www.library.upenn.edu/policies/appropriate-use-policy.html"&gt;Restrictions On Use&lt;/a&gt;.
  &lt;br /&gt;&lt;br /&gt;
  In addition, users agree to the &lt;a href="http://www.upenn.edu/computing/policy/aup.html"&gt;University's Appropriate Use Policy&lt;/a&gt;.
  &lt;/div&gt;
  &lt;/div&gt;&lt;!-- close holder --&gt;
  &lt;/body&gt;
  &lt;/html&gt;
  </code></pre>