Sending back-to-back POST requests using Python
<p>I'm writing a Python script to scrape data from this link: <a href="http://emma.msrb.org/MarketActivity/RecentOS.aspx" rel="nofollow">http://emma.msrb.org/MarketActivity/RecentOS.aspx</a>. I've already configured my script to bypass the confirmation page (using a cookie), and I've figured out how to use a POST request to get to "All New Issues" rather than the default of "Today", but now I need to cycle through all of the 19 pages.</p>

<p>The POST request looks like this: <code>ctl00$gridViewPagingUserControl$page2LinkButton</code>, but using that doesn't get me what I need.</p>

<p>I'm using the Python Requests module, but I've also tried it with urllib/urllib2 and that doesn't seem to work either.</p>

<p>Basically, how can I send multiple/back-to-back POST requests to get to the pages I need?</p>

<p>My sample code is below. Using <code>post_requests_1</code> works fine, but <code>post_requests_2</code> does not.</p>

<pre><code># Cookies
cp = urllib2.HTTPCookieProcessor()
cj = cp.cookiejar
cj.set_cookie(cookielib.Cookie(0, 'Disclaimer', '247245968.1342815275.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)', '80', False, '.emma.msrb.org', True, False, '/', True, False, None, False, None, None, None))

post_requests_1 = { r'__EVENTTARGET' : r'ctl00$mainContentArea$allNewIssuesLinkButton', r'__EVENTARGUMENT' : r'' }
post_requests_2 = { r'__EVENTTARGET' : r'ctl00$mainContentArea$gridViewPagingUserControl$page2LinkButton', r'__EVENTARGUMENT' : r'' }
html = requests.post('http://emma.msrb.org/MarketActivity/RecentOS.aspx', data = post_requests_1, cookies = cj)
print html.text.encode('utf-8')
</code></pre>
