title: Python Requests (Web Scraping) - Building a cookie to be able to view data in a website
body: <p>I'm trying to scrape a finance website to make an application that compares the accuracy of financial data from various other websites (google/yahoo finance).</p>
  <p>The URL I am trying to scrape (specifically the stock's &quot;Key Data&quot; like Market Cap, Volume, Etc) is here:</p>
  <p><a href="https://www.marketwatch.com/investing/stock/sbux" rel="nofollow noreferrer">https://www.marketwatch.com/investing/stock/sbux</a></p>
  <p>I've figured out (with the help of others) that a cookie must be built and sent with each request in order for the page to display the data (otherwise the page html response pretty much returns empty).</p>
  <p>I used Opera/Firefox/Chrome browsers to look into the HTTP Headers and requests that are being sent back from the browser. I've come to the conclusion that there are 3 steps/requests that need to be done to receive all the cookie data and build it piece by piece.</p>
  <p><strong>Step/Request 1</strong></p>
  <p>Simply visiting the above URL.</p>
  <pre><code>GET /investing/stock/sbux HTTP/1.1
  Host: www.marketwatch.com:443
  Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
  Accept-Encoding: gzip, deflate, br
  Accept-Language: en-US,en;q=0.9
  Upgrade-Insecure-Requests: 1
  User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36 OPR/55.0.2994.44
  
  HTTP/1.1 200 OK
  Cache-Control: max-age=0, no-cache, no-store
  Connection: keep-alive
  Content-Length: 579
  Content-Type: text/html; charset=utf-8
  Date: Sun, 26 Aug 2018 05:12:16 GMT
  Expires: Sun, 26 Aug 2018 05:12:16 GMT
  Pragma: no-cache
  </code></pre>
  <p><strong>Step/Request 2</strong></p>
  <p>I am not sure where this &quot;POST&quot; URL came from. However, using Firefox and viewing network connections this url popped up in the &quot;Trace Stack&quot; tab. Again, I have no idea where to get this URL if its the same for everyone or randomly created. I also don't know what POST data is being sent or where the values of X-Hash-Result or X-Token-Value came from. However, this request returns a very important value in the response header with the following line: <strong>'Set-Cookie: ncg_g_id_zeta=701c19ee3f45d07b56b40fb8e313214d'</strong> this piece of the cookie is crucial for the next request in order to return the full cookie and receive the data on the web-page.</p>
  <pre><code>POST /149e9513-01fa-4fb0-aad4-566afd725d1b/2d206a39-8ed7-437e-a3be-862e0f06eea3/fingerprint HTTP/1.1
  Host: www.marketwatch.com:443
  Accept: */*
  Accept-Encoding: gzip, deflate, br
  Accept-Language: en-US,en;q=0.9
  Content-Type: application/json; charset=UTF-8
  Origin: https://www.marketwatch.com
  Referer: https://www.marketwatch.com/investing/stock/sbux
  User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36 OPR/55.0.2994.44
  X-Hash-Result: 701c19ee3f45d07b56b40fb8e313214d
  X-Token-Value: 900c4055-ef7a-74a8-e9ec-f78f7edc363b
  
  HTTP/1.1 200 OK
  Cache-Control: max-age=0, no-cache, no-store
  Connection: keep-alive
  Content-Length: 17
  Content-Type: application/json; charset=utf-8
  Date: Sun, 26 Aug 2018 05:12:16 GMT
  Expires: Sun, 26 Aug 2018 05:12:16 GMT
  Pragma: no-cache
  Set-Cookie: ncg_g_id_zeta=701c19ee3f45d07b56b40fb8e313214d; Path=/; HttpOnly
  </code></pre>
  <p><strong>Step/Request 3</strong></p>
  <p>This request is sent to the original URL with the cookie picked up in step 2. The full cookie is then returned in the response which can be used in step 1 to avoid going through step 2 and 3 again. It will also display the full page of data.</p>
  <pre><code>GET /investing/stock/sbux HTTP/1.1
  Host: www.marketwatch.com:443
  Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
  Accept-Encoding: gzip, deflate, br
  Accept-Language: en-US,en;q=0.9
  Cookie: ncg_g_id_zeta=701c19ee3f45d07b56b40fb8e313214d
  Referer: https://www.marketwatch.com/investing/stock/sbux
  Upgrade-Insecure-Requests: 1
  User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36 OPR/55.0.2994.44
  
  HTTP/1.1 200 OK
  Cache-Control: max-age=0, no-cache, no-store
  Connection: keep-alive
  Content-Encoding: gzip
  Content-Length: 62944
  Content-Type: text/html; charset=utf-8
  Date: Sun, 26 Aug 2018 05:12:17 GMT
  Expires: Sun, 26 Aug 2018 05:12:17 GMT
  Pragma: no-cache
  Server: Kestrel
  Set-Cookie: seenads=0; expires=Sun, 26 Aug 2018 23:59:59 GMT; domain=.marketwatch.com; path=/
  Set-Cookie: mw_loc=%7B%22country%22%3A%22CA%22%2C%22region%22%3A%22ON%22%2C%22city%22%3A%22MARKHAM%22%2C%22county%22%3A%5B%22%22%5D%2C%22continent%22%3A%22NA%22%7D; expires=Sat, 01 Sep 2018 23:59:59 GMT; domain=.marketwatch.com; path=/
  Vary: Accept-Encoding
  x-frame-options: SAMEORIGIN
  x-machine: 8cfa9f20bf3eb
  </code></pre>
  <p><strong>Summary</strong></p>
  <p>In summary, step 2 is the most important to get the remaining cookie piece... But I can't figure out the 3 things:</p>
  <p><strong>1)</strong> Where the POST url comes from (not embedded in original page, is the URL the same for everyone or is it randomly generated by the site).</p>
  <p><strong>2)</strong> What's the data being sent in the POST request?</p>
  <p><strong>3)</strong> Where do X-Hash-Result and X-Token-Value come from? Is it required to be sent in the header with the request?</p>