title: Extracting useful/readable content from a website
body: <p>I am working on a application that needs to scrape a part of a website the user submits. I want to collect useful and readable content from the website and definitely not the whole site. If I look at applications that also do this (<a href="http://thinkery.me" rel="nofollow">thinkery</a> for example) I notice that that they somehow managed to create a way to scrape the website, try to guess what useful content is, show it in a readable format and they do that pretty fast.</p>
  
  <p>I've been playing with cURL and I am getting pretty near the result I want but I have some issues and was wondering if someone has some more insights.</p>
  
  <pre><code>    $ch = curl_init('http://www.example.org');
      curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
      curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  
      // $content contains the whole website
      $content = curl_exec($ch);
  
      curl_close($ch);
  </code></pre>
  
  <p>With the very simple code above I can scrape the whole website and with preg_match() I can try to find div's with the class, id or properties which contains the string 'content', 'summary' et cetera.</p>
  
  <p>If preg_match() has result I can fairly guess that I have found relevant content and save this as the summary of the saved page. The problem I have is that cURL saves the WHOLE page in memory so this can take up a lot of time and resources. And I think that doing a preg_match() over such a large result can also take up a lot of time.</p>
  
  <p>Is there a better way to achieve this? </p>