title: How To Aggregate API Data?
body: <p>I have a system that connects to 2 popular APIs. I need to aggregate the data from each into a unified result that can then be paginated. The scope of the project means that the system could end up supporting 10's of APIs.</p>
  
  <p>Each API imposes a max limit of 50 results per request.</p>
  
  <p>What is the best way of aggregating this data so that it is reliable i.e ordered, no duplicates etc</p>
  
  <p>I am using CakePHP framework on a LAMP environment, however, I think this question relates to all programming languages.</p>
  
  <p>My approach so far is to query the search API of each provider and then populate a MySQL table. From this the results are ordered, paginated etc. However, my concern is performance: API communication, parsing, inserting and then reading all in one execution.</p>
  
  <p>Am I missing something, does anyone have any other ideas? I'm sure this is a common problem with many alternative solutions.</p>
  
  <p>Any help would be greatly appreciated.</p>