title: What's the fastest way to scrape a lot of pages in php?
body: <p>I have a data aggregator that relies on scraping several sites, and indexing their information in a way that is searchable to the user.</p>
  
  <p>I need to be able to scrape a vast number of pages, daily, and I have ran into problems using simple curl requests, that are fairly slow when executed in rapid sequence for a long time (the scraper runs 24/7 basically).</p>
  
  <p>Running a multi curl request in a simple while loop is fairly slow. I speeded it up by doing individual curl requests in a background process, which works faster, but sooner or later the slower requests start piling up, which ends up crashing the server. </p>
  
  <p>Are there more efficient ways of scraping data? perhaps command line curl?</p>