title: How do I scrape pages with dynamically generated URLs using Python?
body: <p>I am trying to scrape <a href="http://www.dailyfinance.com/quote/NYSE/international-business-machines/IBM/financial-ratios" rel="noreferrer">http://www.dailyfinance.com/quote/NYSE/international-business-machines/IBM/financial-ratios</a>, but the traditional url string building technique doesn't work because the "full-company-name-is-inserted-in-the-path" string.  And the exact "full-company-name" isn't known in advance. Only the company symbol, "IBM" is known.</p>
  
  <p>Essentially, the way I scrape is by looping through an array of company symbol and build the url string before sending it to urllib2.urlopen(url).  But in this case, that can't be done.</p>
  
  <p>For example, CSCO string is </p>
  
  <pre><code>http://www.dailyfinance.com/quote/NASDAQ/cisco-systems-inc/CSCO/financial-ratios
  </code></pre>
  
  <p>and another example url string is AAPL:</p>
  
  <pre><code>http://www.dailyfinance.com/quote/NASDAQ/apple/AAPL/financial-ratios
  </code></pre>
  
  <p>So in order to get the url, I had to search the symbol in the input box on the main page: </p>
  
  <pre><code>http://www.dailyfinance.com/
  </code></pre>
  
  <p>I've noticed that when I type "CSCO" and inspect the search input at (<a href="http://www.dailyfinance.com/quote/NASDAQ/apple/AAPL/financial-ratios" rel="noreferrer">http://www.dailyfinance.com/quote/NASDAQ/apple/AAPL/financial-ratios</a> in Firefox web developer network tab, I noticed that the get request is sending to </p>
  
  <pre><code>http://j.foolcdn.com/tmf/predictivesearch?callback=_predictiveSearch_csco&amp;term=csco&amp;domain=dailyfinance.com
  </code></pre>
  
  <p>and that the referer actually gives the path that I want to capture</p>
  
  <pre><code>Host: j.foolcdn.com
  User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0
  Accept: */*
  Accept-Language: en-US,en;q=0.5
  Accept-Encoding: gzip, deflate
  Referer: http://www.dailyfinance.com/quote/NASDAQ/cisco-systems-inc/CSCO/financial-ratios?source=itxwebtxt0000007
  Connection: keep-alive
  </code></pre>
  
  <p>Sorry for the long explanation.  So the question is how do I extract the url in the Referer?  If that is not possible, how should I approach this problem?  Is there another way?</p>
  
  <p>I really appreciate your help.</p>