title: How to speed up GET requests when using a Queue
body: <p>I'm using Huey/Hueyx as a queue to buffer resources I need to GET with pythons requests library. Credentials (a token) are stored in Redis and the task executor calls the function, gets the resource and then indexes it to elastic search. Because I need to request millions of different resources and my solution is quite slow (around 5k requests per minute) the queue backs up very quickly and the performance takes a hit.</p>
  
  <pre><code>def get_raw_resource(url, token, id):
          try:
                  url = f"{url}/api/v2/{id}"
  
                  headers = {
                      'Accept': "application/json",
                      'Authorization': f"Bearer {token}",
                      'Cache-Control': "no-cache"
                  }
                  response = requests.get(url, headers=headers) 
          [snip]
  
  
  @res_q.task(delay=10, retries=3, retry_delay=180)
  def get_resource_es(client_id, id):
          [snipped redis stuff]
          try:
                  res = get_raw_resource(url, token, id)
                  parsed = parse_res(res)
                  es.index(index=es_index, body=parsed, id=id)
          except Exception as e:
                  raise e
  </code></pre>
  
  <p>My question is: how can I make this whole ordeal faster with some easy steps? Adding more consumers is always an option, but it's not something I want to do right now before exploring other options. I've read about async code, however it looks like it would quadruple the complexity of the code. Also, I know nothing about the async programming model.</p>