title: Web Scraping API - "See the scroll api for a more efficient way to request large data sets."
body: <p>I am trying to scrape the website, <a href="https://www.huntr.co/jobs" rel="nofollow noreferrer">https://www.huntr.co/jobs</a> and it has about 1 million listing. I checked and found that the website is ajax loaded so decided to scrape the website using its API which is, <a href="https://app.huntr.co/public/search/job-posts?page=1" rel="nofollow noreferrer">https://app.huntr.co/public/search/job-posts?page=1</a>. The website has data loaded in json format and when I change &quot;page= &quot; I get a new page with new data. But whenever I cross &quot;page = 332&quot;, the API gives the following message in json:</p>
  <pre><code>{&quot;message&quot;:&quot;[query_phase_execution_exception] Result window is too large, from + size must be less 
  than or equal to: [10000] but was [30030]. See the scroll api for a more efficient way to request 
  large data sets. This limit can be set by changing the [index.max_result_window] index level 
  setting.&quot;,&quot;error&quot;:{&quot;status&quot;:500}
  </code></pre>
  <p>I dont understand what a scroll api is and how can I request that. Due to this error I am able to scrape about only 10,000 listing out of 1 million listing. I am using python for this project. Any solution will be highly appreciated !</p>