title: How to make nutch crawl files and subfolders - it only crawls the index of the folder
body: <p>EDIT: I found my answer and wrote it up below, but gave the bounty to tahagh, since he provided some good suggestions.</p>
  
  <hr>
  
  <p>I am setting up nutch to crawl a local folder (a samba mount). I have followed <a href="http://amac4.blogspot.dk/2013/07/setting-up-nutch-to-crawl-filesystem.html" rel="nofollow">this</a> tutorial. </p>
  
  <p>My folder looks like this:</p>
  
  <pre><code>nutch@ubuntu:~$ ls /mnt/ntserver/
  expansion.docx  test-folder  test-shared.txt
  </code></pre>
  
  <p>with some files and folders below <code>test-folder</code> also.</p>
  
  <p>When I run nutch, it doesn't index the files or the subfolder. It only puts a single document into solr, which is the index of the folder. This is what I get in solr after running nutch on an empty solr index:</p>
  
  <pre><code>"response": {
      "numFound": 1,
      "start": 0,
      "docs": [
        {
          "content": [
            "Index of /mnt/ntserver Index of /mnt/ntserver ../ - - - expansion.docx Mon, 30 Dec 2013 14:00:42 GMT 70524 test-folder/ Fri, 17 Jan 2014 09:38:50 GMT - test-shared.txt Thu, 16 Jan 2014 11:33:42 GMT 16"
          ],
        .....
  </code></pre>
  
  <p>How can I get nutch to index the files and the subfolders?</p>
  
  <hr>
  
  <p>edit: if I set regex-urlfilter to allow everything (after filtering for gifs, http etc) like this <code>+.</code>, then nutch seems to go up the folder hierarchy, but not down, and still only crawling the indexes, not the files. This is what I get in solr:</p>
  
  <pre><code>"response": {
      "numFound": 26,
      "start": 0,
      "docs": [
        {
          "title": [
            "Index of /"
          ]
        },
        {
          "title": [
            "Index of /bin"
          ]
        },
        ...
        {
          "title": [
            "Index of /mnt"
          ]
        },
        {
          "title": [
            "Index of /mnt/ntserver"
          ]
        },
        ...
      ]
  </code></pre>
  
  <hr>
  
  <p>Additional info:</p>
  
  <p>This is the crawl command I use:</p>
  
  <pre><code>apache-nutch-1.7/bin/nutch crawl -dir fileCrawl -urls apache-nutch-1.7/urls/ -solr http://localhost:8983/solr -depth 3 -topN 10000
  </code></pre>
  
  <p>This is the content of my seed urls file:</p>
  
  <pre><code>nutch@ubuntu:~$ cat apache-nutch-1.7/urls/urls_to_be_crawled.txt 
  file:////mnt/ntserver
  </code></pre>
  
  <p>this is my regex-urlfilter.xml:</p>
  
  <pre><code>nutch@ubuntu:~$ cat apache-nutch-1.7/conf/regex-urlfilter.txt
  # skip http: ftp: and mailto: urls
  -^(http|ftp|mailto):
  
  # skip image and other suffixes we can't yet parse
  # for a more extensive coverage use the urlfilter-suffix plugin
  -\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|wmf|WMF|zip|ZIP|ppt|PPT|mpg|MPG|gz|GZ|rpm|RPM|tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS|asp|ASP|xxx|XXX|yyy|YYY|cs|CS|dll|DLL|refresh|REFRESH)$
  
  # accept any files
  +.*mnt/ntserver.*
  </code></pre>
  
  <p>I have included <code>protocol-file</code> and set no limit on file size in nutch-site.xml:</p>
  
  <pre><code>nutch@ubuntu:~$ cat apache-nutch-1.7/conf/nutch-site.xml
  ...
  &lt;property&gt;
      &lt;name&gt;plugin.includes&lt;/name&gt;
      &lt;value&gt;protocol-file|urlfilter-regex|parse-(html|tika|text)|index-(basic|anchor)|indexer-solr|scoring-opic|urlnormalizer-(pass|regex|basic)|index-more&lt;!--|remove-empty-document|title-adder--&gt;&lt;/value&gt;
      &lt;description&gt;&lt;/description&gt;
  &lt;/property&gt;
  
  &lt;property&gt;
      &lt;name&gt;file.content.limit&lt;/name&gt;
      &lt;value&gt;-1&lt;/value&gt;
      &lt;description&gt; Needed to stop buffer overflow errors - Unable to read.....&lt;/description&gt;
  &lt;/property&gt;
  
  ...
  </code></pre>
  
  <p>and I have commented out the duplicate slash removal in regex-normalize.xml:</p>
  
  <pre><code>nutch@ubuntu:~$ cat apache-nutch-1.7/conf/regex-normalize.xml
  ...
  &lt;!-- removes duplicate slashes - commented out, so we won't get invalid filenames 
  &lt;regex&gt;
      &lt;pattern&gt;(?&amp;lt;!:)/{2,}&lt;/pattern&gt;
      &lt;substitution&gt;/&lt;/substitution&gt;
  &lt;/regex&gt;
  --&gt;
  ...
  </code></pre>