title: Python2.7 Get the content of an iframe with requests
body: <p><strong>How do I get the content of this iframe with python's requests?</strong></p>
  
  <p>In Firebug, the content of the iframe is in a response from a POST request which I am finding difficult to access using Python. </p>
  
  <p><strong>Code.</strong></p>
  
  <pre><code>import requests
  
  iframe_url = "https://wwwrs.resbank.co.za/BpsReports/InstNotice.aspx"
  
  # How do I scrape the payload for the POST request?
  r = requests.post(iframe_url) # data=?
  
  r.text
  </code></pre>
  
  <p><strong>The response.</strong></p>
  
  <pre><code>&lt;!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"&gt;
  &lt;html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"&gt;
  &lt;head&gt;&lt;title&gt;Bank Institutions&lt;/title&gt;...
  </code></pre>
  
  <p><strong>The response I want.</strong></p>
  
  <pre><code>1|#||4|224|updatePanel|ctl00_MainContent_ReportViewer1_DocMap|&lt;div id="ctl00_MainContent_ReportViewer1_ctl09" style="display:none;"&gt;
  &lt;input type="hidden" name="ctl00$MainContent$ReportViewer1$ctl09$ClientClickedId" id="ctl00_MainContent_ReportViewer1_ctl09_ClientClickedId" /&gt;&lt;/div&gt;...
  </code></pre>
  
  <p>How do I scrape the data for the POST request?</p>
  
  <p><strong>EDIT.</strong></p>
  
  <p>I have added browser headers and a payload in the POST request. The response looks like the one I want (inasmuch as it is pipe-delimited) but it doesn't show the content of the iframe. </p>
  
  <p><strong>Additional code - Headers and payload.</strong></p>
  
  <pre><code>headers = {
      'User-agent': 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1', 
      'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'
  }
  
  payload = {
      "ctl00$MainContent$ScriptManager1": "ctl00$MainContent$ScriptManager1|ctl00$MainContent$ReportViewer1$ctl10$Reserved_AsyncLoadTarget",
      "__EVENTTARGET": "ctl00$MainContent$ReportViewer1$ctl10$Reserved_AsyncLoadTarget",
      "__EVENTARGUMENT": "",
      "__VIEWSTATE":  
      "/wEPDwUKLTYyOTE1Njg2OQ9kFgJmD2QWA..." # shortened 
      "__VIEWSTATEGENERATOR": "E6F8CED1",
      "__EVENTVALIDATION":    
      "/wEdABKvYwmAyl699cXAjRyhDjs8urvg...", # shortened
      "ctl00$MainContent$ReportViewer1$ctl03$ctl11":"ltr",
      "ctl00$MainContent$ReportViewer1$ctl03$ctl12":"standards",
      #"ctl00$MainContent$ReportViewer1$AsyncWait$HiddenCancelField":"False",
      "ctl00$MainContent$ReportViewer1$ctlToggleParam$store":"false",
      #"ctl00$MainContent$ReportViewer1$ctl08$collapse":"false",
      "ctl00$MainContent$ReportViewer1$ctl10$VisibilityState$ctl00":None,
      #"ctl00$MainContent$ReportViewer1$ctl10$ReportControl$ctl04":100,
      "__ASYNCPOST": "true"
  }
  
  r = requests.post(url, headers=headers, data=payload, timeout=3.5)
  </code></pre>
  
  <p>Several keys had no value and a few of the key-value pairs were causing a 500 server error so I have omitted them and commented them out respectively.</p>
  
  <p><strong>The response in Python does not include the iframe.</strong></p>
  
  <pre><code>1|#||4|7645|updatePanel|ctl00_MainContent_ReportViewer1_ReportViewer|
    &lt;div id="ctl00_MainContent_ReportViewer1" 
    onclick="if 
     ($get('ctl00_MainContent_ReportViewer1_ctl04') != null &amp;&amp; 
      $get('ctl00_MainContent_ReportViewer1_ctl04').control != null) 
      $get('ctl00_MainContent_ReportViewer1_ctl04').control.HideActiveDropDown();" 
    onactivate="if
     ($get('ctl00_MainContent_ReportViewer1_ctl04') != null &amp;&amp; 
      $get('ctl00_MainContent_ReportViewer1_ctl04').control != null) 
      $get('ctl00_MainContent_ReportViewer1_ctl04').control.HideActiveDropDown();" style="height:400px;width:400px;position:absolute;left:0px;Top:0px"&gt;
    &lt;div id="ctl00_MainContent_ReportViewer1_HttpHandlerMissingErrorMessage" style="border-color:Red;border-width:2px;border-style:Solid;padding:10px;display:none;overflow:auto;font-size:.85em;"&gt;
     &lt;h2&gt;
        Report Viewer Configuration Error
     &lt;/h2&gt;&lt;p&gt;The Report Viewer Web Control HTTP Handler has not been registered in the application's web.config file.  
              Add &lt;add verb="*" path="Reserved.ReportViewerWebControl.axd" 
              type = "Microsoft.Reporting.WebForms.HttpHandler, Microsoft.ReportViewer.WebForms, 
              Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" /&gt; to the system.web/httpHandlers section of the web.config file, or add &lt;add 
              name="ReportViewerWebControlHandler" preCondition="integratedMode" verb="*" path="Reserved.ReportViewerWebControl.axd"
              type="Microsoft.Reporting.WebForms.HttpHandler, Microsoft.ReportViewer.WebForms, 
              Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" /&gt; to the system.webServer/handlers section for Internet Information Services 7 or later.
          &lt;/p&gt;
    &lt;/div&gt;
    &lt;span id="ctl00_MainContent_ReportViewer1_ctl03"&gt;
      &lt;input type="hidden" name="ctl00$MainContent$ReportViewer1$ctl03$ctl00" id="ctl00_MainContent_ReportViewer1_ctl03_ctl00" /&gt;
      &lt;input type="hidden" name="ctl00$MainContent$ReportViewer1$ctl03$ctl01" id="ctl00_MainContent_ReportViewer1_ctl03_ctl01" /&gt;
    &lt;/span&gt;
      &lt;input type="hidden" name="ctl00$MainContent$ReportViewer1$ctl11" id="ctl00_MainContent_ReportViewer1_ctl11" /&gt;
      &lt;input type="hidden" name="ctl00$MainContent$ReportViewer1$ctl12" id="ctl00_MainContent_ReportViewer1_ctl12" /&gt;
        &lt;div id="ctl00_MainContent_ReportViewer1_AsyncWait" style="background-color:White;opacity:0.7;position:absolute;display:none;filter:alpha(opacity=70);"&gt;
        &lt;/div&gt;
        &lt;div id="ctl00_MainContent_ReportViewer1_AsyncWait_Wait" style="cursor:wait;background-color:#ECE9D8;padding:15px;border:1px solid black;display:none;position:absolute;"&gt;
          &lt;table height="100%"&gt;
            &lt;tr&gt;
              &lt;td width="32px" height="32px"&gt;
                &lt;img src="/BpsReports/Reserved.ReportViewerWebControl.axd?OpType=Resource&amp;Version=10.0.30319.1&amp;Name=Microsoft.Reporting.WebForms.Icons.SpinningWheel.gif" style="height:32px;width:32px;" /&gt;
              &lt;/td&gt;
              &lt;td style="vertical-align:middle;text-align:center;"&gt;
                &lt;span style="font-family:Verdana;font-size:14pt;"&gt;Loading...&lt;/span&gt;
                  &lt;div style="margin-top:3px;"&gt;
                    &lt;a href="javascript:$get('ctl00_MainContent_ReportViewer1_AsyncWait').control._cancelCurrentPostback();" style="font-family:Verdana;font-size:8pt;color:#3366CC;"&gt;Cancel&lt;/a&gt;...
  </code></pre>
  
  <p><strong>The response in Firebug includes the iframe.</strong></p>
  
  <pre><code>1|#||4|224|updatePanel|ctl00_MainContent_ReportViewer1_DocMap|&lt;div id="ctl00_MainContent_ReportViewer1_ctl09"
   style="display:none;"&gt;
      &lt;input type="hidden" name="ctl00$MainContent$ReportViewer1$ctl09$ClientClickedId" id="ctl00_MainContent_ReportViewer1_ctl09_ClientClickedId"
   /&gt;
  &lt;/div&gt;|5472|updatePanel|ctl00_MainContent_ReportViewer1_ctl10_ReportArea|&lt;div NewContentType="Microsoft
  .Reporting.WebFormsClient.ReportAreaContent.ReportPage" ForNonReportContentArea="false" id="ctl00_MainContent_ReportViewer1_ctl10_VisibilityState"
   style="visibility:none;"&gt;
      &lt;input type="hidden" name="ctl00$MainContent$ReportViewer1$ctl10$VisibilityState$ctl00" value="ReportPage"
   /&gt;
  &lt;/div&gt;&lt;input type="hidden" name="ctl00$MainContent$ReportViewer1$ctl10$ScrollPosition" id="ctl00_MainContent_ReportViewer1_ctl10_ScrollPosition"
   /&gt;&lt;span id="ctl00_MainContent_ReportViewer1_ctl10_Reserved_AsyncLoadTarget"&gt;&lt;/span&gt;&lt;div id="ctl00_MainContent_ReportViewer1_ctl10_ReportControl"
   style="display:none;"&gt;
      &lt;span&gt;&lt;/span&gt;&lt;input type="hidden" name="ctl00$MainContent$ReportViewer1$ctl10$ReportControl$ctl02" id
  ="ctl00_MainContent_ReportViewer1_ctl10_ReportControl_ctl02" /&gt;&lt;input type="hidden" name="ctl00$MainContent$ReportViewer1$ctl10$ReportControl$ctl03"
   id="ctl00_MainContent_ReportViewer1_ctl10_ReportControl_ctl03" /&gt;&lt;input type="hidden" name="ctl00$MainContent$ReportViewer1$ctl10$ReportControl$ctl04"
   id="ctl00_MainContent_ReportViewer1_ctl10_ReportControl_ctl04" value="100" /&gt;&lt;div style="display:none
  ;"&gt;
          &lt;DIV dir=...The content of the iframe.
  </code></pre>
  
  <p><strong>Questions which may aid a solution.</strong></p>
  
  <ol>
  <li><p>How do I scrape the payload for the POST request? </p></li>
  <li><p>Is the sequence of requests preparing the payload?  </p></li>
  <li><p>How do I get the iframe with requests?</p></li>
  </ol>
  
  <p>Thanks for reading.</p>