title: Is there a way to leverage Hadoop tools to mange parallel REST API calls to external sources?
body: <p>I am writing software that creates a large graph database.  The software needs to access dozens of different REST APIs with millions of total requests.  The data will then be processed by the Hadoop cluster.  Each of these APIs have rate limits that vary by requests/second, per window, per day and per user (typically via OAuth).</p>
  
  <p>Does anyone have any suggestions on how I might use either a Map function or other Hadoop-ecosystem tool to manage these queries?  The goal would to be to leverage the parallel processing in Hadoop.</p>
  
  <p>Because of the varied rate limits, it often makes sense to switch to a different API query while waiting for the first limit to reset.  An example would be one API call that creates nodes in the graph and another that enriches the data for that node.  I could have the system go out and enrich the data for the new nodes while waiting for the first API limit to reset.</p>
  
  <p>I have tried using SQS queuing on EC2 to manage the various API limits and states (creating a queue for each API call), but have found it to be ridiculously slow.  </p>
  
  <p>Any ideas?</p>