title: Batched API call inside apache spark?
body: <p>I am a beginner to Apache Spark and I do have the following task:</p>
  
  <p>I am reading records from a datasource that - within the spark transformations - need to be enhanced by data from a call to an external webservice before they can be processed any further. </p>
  
  <p>The webservice will accept parallel calls to a certain extent, but only allows a few hundred records to be sent at once. Also, it's quite slow, so batching up as much as possible and parallel requests are definitely helping here.</p>
  
  <p>Is there are way to do this with spark in a reasonable manner? </p>
  
  <p>I thought of reading records, pre-process them to another datasource, then read the "API-Queue" data source 500 records at a time (if possible with multiple processes) and write the records to the next datasource, and use this result datasource to do the final transformations. </p>
  
  <p>The only place where those weird limits need to be respected is within the API calls (that's why I thought some intermediate data format / data source would be appropriate). </p>
  
  <p>Any ideas or directions you want to point me to?</p>