title: How to get historical data of question and answers from stackoverlfow exchange without being throttled?
body: <p>I am trying to read all AAD related questions and answers from Stack Exchange API <a href="https://api.stackexchange.com/2.2/search/advanced/pagesize=100&amp;fromdate=2019-07-01&amp;todate=2020-10-19&amp;site=stackoverflow&amp;filter=!BLIw93LDFyFBUjlepdSTkMo7r6Pkpx&amp;q=listOfTags"><code>/2.2/search/advanced/pagesize=100&amp;fromdate=2019-07-01&amp;todate=2020-10-19&amp;site=stackoverflow&amp;filter=!BLIw93LDFyFBUjlepdSTkMo7r6Pkpx&amp;q=listOfTags</code></a> by passing set of tags, since we are trying to get the data from July 1st 2019.</p>
  <p>Our ADF pipeline keeps getting throttlede and even if we set the wait time for 1 minute and our ETL is very slow, it's running forever.</p>
  <p>Current Approach (very slow)</p>
  <ol>
  <li><p>I am using ADF to Pull the all the questions (iterating through page by page using until activity) which meets the tags and load the data into SQL
  <a href="https://i.stack.imgur.com/qZyAR.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/qZyAR.png" alt="enter image description here" /></a></p>
  </li>
  <li><p>Pass the question id to this API <a href="https://api.stackexchange.com/docs/answers-on-questions#order=desc&amp;sort=activity&amp;ids=29433422&amp;filter=!0U7YRMKgNJq(Exonzn(PdiZE5&amp;site=stackoverflow&amp;run=true">https://api.stackexchange.com/docs/answers-on-questions#order=desc&amp;sort=activity&amp;ids=29433422&amp;filter=!0U7YRMKgNJq(Exonzn(PdiZE5&amp;site=stackoverflow&amp;run=true</a> to get all the answers for respective question and then load the result into SQL.</p>
  </li>
  </ol>
  <p>Questions:</p>
  <ol>
  <li><p>Is there a direct back-end (Kusto or SQL or cosmos etc.) we can get the data than calling the API to get the question and answers? If so how do we get the access to the back-end?</p>
  </li>
  <li><p>What is the efficient approach to pull the historical data without throttling from Stack Overflow?</p>
  </li>
  </ol>