title: Which web crawler to use to save news articles from a website into .txt files?
body: <p>i am currently in dire need of news articles to test a LSI implementation (it's in a foreign language, so there isnt the usual packs of files ready to use).</p>
  
  <p>So i need a crawler that given a starting url, let's say <a href="http://news.bbc.co.uk/" rel="nofollow noreferrer">http://news.bbc.co.uk/</a> follows all the contained links and saves their content into .txt files, if we could specify the format to be UTF8 i would be in heaven.</p>
  
  <p>I have 0 expertise in this area, so i beg you for some sugestions in which crawler to use for this task.</p>