title: How to extend Nutch for article crawling
body: <p>I'm look for a framework to grab articles, then I find Nutch 2.1. Here's my plan and questions in each:</p>
  
  <p>1</p>
  
  <p>Add article list pages into url/seed.txt
      Here's one problem. What I actually want to be indexed is the article pages, not the article list pages. But, if I don't allow the list page to be indexed, Nutch will do nothing because the list page is the entrance. So, how can I index only the article page without list pages?</p>
  
  <p>2</p>
  
  <p>Write a plugin to parse out the 'author', 'date', 'article body', 'headline' and maybe other information from html.
      The 'Parser' plugin interface in Nutch 2.1 is:
      Parse getParse(String url, WebPage page)
      And the 'WebPage' class has some predefined attributs:</p>
  
  <pre><code>public class WebPage extends PersistentBase {
    // ...
    private Utf8 baseUrl;
    // ...
    private ByteBuffer content; // &lt;== This becomes null in IndexFilter
    // ...
    private Utf8 title;
    private Utf8 text;
    // ...
    private Map&lt;Utf8,Utf8&gt; headers;
    private Map&lt;Utf8,Utf8&gt; outlinks;
    private Map&lt;Utf8,Utf8&gt; inlinks;
    private Map&lt;Utf8,Utf8&gt; markers;
    private Map&lt;Utf8,ByteBuffer&gt; metadata;
    // ...
  }
  
  So, as you can see, there are 5 maps I can put my specified attributes in. But, 'headers', 'outlinks', 'inlinks' seem not used for this. Maybe I could put those information into markers or metadata. Are they designed for this purpose?
  BTW, the Parser in trunk looks like: 'public ParseResult getParse(Content content)', and seems more reasonable for me.
  </code></pre>
  
  <p>3</p>
  
  <p>After the articles are indexed into Solr, another application can query it by 'date' then store the article information into Mysql.
      My question here is: can Nutch store the article directly into Mysql? Or can I write a plugin to specify the index behavior?</p>
  
  <p>Is Nutch a good choice for my purpose? If not, do you guys suggest another good quality framework/library for me?
  Thanks for your help.</p>