title: How to find the Request URL from a webpage in chromes's Inspect Element> Network > XHR > Response > Headers > General
body: <p>Sorry if I don't know the correct terminology, I'm new to web scraping, please correct my terminology if you feel like it.</p>
  <p>I'm working on a project to scrape images off all the pieces by an artist given the URL for the artist's gallery. What I am doing is finding the unique id of each page of the gallery that will lead me to the webpage hosting the original image. I can already scrape from the art page I just need the id's of each page from gallery.</p>
  <p>Artist Gallery --&gt; Art Page --&gt; scrape image</p>
  <p>The id's of each page on the gallery page is not available in the page source, since it is being loaded in separately through JavaScript I think, so I can not grab them using:</p>
  <pre><code>import requests
  import urllib.request
  
  
  response = requests.get(pageurl)
  print(response.text)
  </code></pre>
  <p>But I have found that by going to Chrome Inspect Element&gt; Network &gt; XHR &gt; Response &gt; Headers &gt; General, there is a request URL that has all the id's that I need, and below that is a Query String Parameters section that has all the id's I need.
  <a href="https://i.stack.imgur.com/qUdqb.png" rel="nofollow noreferrer">Picture of Query String Parameters</a></p>
  <p><a href="https://i.stack.imgur.com/odjmd.png" rel="nofollow noreferrer">Picture of Request URL</a></p>
  <p>I am using BeautifulSoup, but the problem just lies with how to get the data.
  I have also used <code>urllib.request.urlopen(pageurl)</code> with similar results. I have also tried Selenium, but was still unable to get the ids, although I may not have done so correctly, I was able to get to the webpage, but maybe I did not use the right method. For now, this is what I want to try. EDIT: I have since figured it out using Selenium. (I just wasn't trying hard enough), but would still like some input regarding intercepting XHR's.</p>
  <p><a href="https://www.pixiv.net/member_illust.php?id=11764388&amp;type=illust" rel="nofollow noreferrer">Link to site if you really want to see it, but you may have to login</a></p>