title: Fastest way to calculate the last working page for scraping
body: <p>I'm trying to scrape a website and I don't want to be forced to recursively scrape every page until the "end".</p>
  
  <p>I want to be able to just get the last page, or any page in between.</p>
  
  <p>This isn't an issue most of the time, but this webpage is different. For example:</p>
  
  <p>I do a search query on: "Logitech" </p>
  
  <p>The website returns the following information I want to use:</p>
  
  <ul>
  <li>Total amount of products found: <strong>15.000</strong></li>
  <li>Products per page: <strong>30</strong></li>
  <li>Pagination: <strong>1, 2, 3, 4, ...500</strong> <em>(15.000/30 = 500)</em></li>
  </ul>
  
  <p>However it seems like the <strong>Total amount of products found</strong> is false information, and so is the pagination, because when I try to access <code>https://website.com/products?q=Logitech&amp;page=500</code> there are no products. This also applies on lower values like: 499, 498, 450, 400, 350, etc.</p>
  
  <p>So I tried "calculating" possible page numbers in my head, and I came down to <strong>the actual last page number: 166</strong></p>
  
  <p>I "calculated" this by checking if there were results: </p>
  
  <ul>
  <li>If there were results: increment value by amount x </li>
  <li>If there were no results: decrement value by amount x</li>
  </ul>
  
  <p>So I made a <strong>test script</strong> to sort of replicate my method of finding the page number (repl.it link):
  <a href="https://repl.it/@SaltyPotato1/LastPageCalculator" rel="nofollow noreferrer">https://repl.it/@SaltyPotato1/LastPageCalculator</a></p>
  
  <p>But I have two problems with his script (and I hope you guys can help me out):</p>
  
  <ol>
  <li><p>This script "knows" the right value (<strong>166</strong>), so it can use <code>&gt;</code> and <code>&lt;</code> operators, when I'm scraping; It can only check if there is a <strong>result</strong> in this case: products. This means that there are only 2 options:</p>
  
  <ul>
  <li>If there is a result it will mean that: the page number is either <strong>too low</strong> <strong>or</strong> it's in fact the last page number.</li>
  <li>If there is <strong>no result</strong> it will mean that: the page number is <strong>too</strong> high.</li>
  </ul></li>
  <li><p>I have a feeling the script/calculating could be improved, since some values are emitted twice.</p></li>
  </ol>
  
  <p>What can I do to determine if it is in fact the <strong>last page number</strong> instead of a page number "under" the actual last page. And how could I minimize the amount of requests that are going to be made to calculate this.</p>
  
  <p>I'm looking forward to your answers/help!</p>