CMS, wikis, text files … modern companies store important data in many different places, and that data must be accessible down to the tiniest detail through a single search. Commercial software vendors such as Google  offer tools that will index the data and store the index on an external server. But many organizations prefer to keep control of the search capabilities – for security and privacy reasons, but also to add flexibility and promote innovation and customization.
A handy constellation of open source tools from the Apache project will help you build your own search index for the assorted documents and data on your network: Nutch, Solr, Apache, and Lucene.
Nutch  is a powerful web crawler, and Apache Solr  is a search engine based on Apache Lucene . You can combine Nutch with Solr to create a complete search engine – a miniature Google, if you like.
The crawler indexes data accessible to the daemons associated with the process. Depending on your security system, the search results could be more than you would want non-privileged users to see, so you might need to adjust your configuration to rule out access to highly secure files and directories.
This workshop shows how to build your own search engine using on an Ubuntu 14.04.2 LTS system.
On Canonical's Enterprise Linux, Solr is available from the package sources; you only need to install Nutch manually (Listing 1, lines 1-4). Then back up Solr's default XML schema and replace it with the file supplied by Nutch (Lines 6 and 7).
line to "true"; then restart Tomcat by typing service tomcat6 restart.
Although you can control the crawler's default behavior with the /opt/nutch/conf/nutch-default.xml file, it makes more sense to customize the /opt/nutch/conf/nutch-site.xml file with site-specific details.
The example in Listing 2 shows how you can configure the name of the HTTP agent. This name will appear in the web server's logfiles.
It makes sense to disable this value and only re-enable it if problems occur.
Another useful variable, fetcher.threads.fetch, defines the number of concurrent threads reading content. The http.timeout reduces the time the thread needs to wait for a request to time out.