title: importing mongodb data to solr in optimal way
body: <p>I have millions of documents in my mongodb database and to implement searching I am using Apache Solr. I googled how to import data from mongodb to solr, but found that there is no straight forward approach using Data Import Handler. So I decided to insert the document into Solr at the same time when I am inserting it into mongoDB using the solr client for Java, SolrJ. My documents are in following format in mongoDB:</p>
  
  <pre><code>{
      "_id" : ObjectId("51cc52c9e4b04f75b27542ba"),
      "article" : {
          "summary" : "As more people eschew landlines, companies are offering technologies like\npersonal cell sites and signal boosters to augment cellphones, Eric A. Taub\nwrites in The New York Times.\n\n",
          "author" : "By THE NEW YORK TIMES",
          "title" : "Daily Report: Trying to Fix a Big Flaw in Cellphone Technology (NYT)",
          "source" : "NYT",
          "publish_date" : "Thu, 27 Jun 2013 12:01:00 +0100",
          "source_url" : "http://bits.blogs.nytimes.com/feed/",
          "url" : "http://news.feedzilla.com/en_us/stories/top-news/315578403?client_source=api&amp;format=json"
      },
      "topics" : [
          {
              "categoryName" : "Technology Internet",
              "score" : "94%"
          }
      ],
      "socialTags" : [
          {
              "originalValue" : "Cell site",
              "importance" : "1"
          },
          {
              "originalValue" : "Cellular repeater",
              "importance" : "1"
          },
          {
              "originalValue" : "Technology Internet",
              "importance" : "1"
          }
      ],
      "entities" : [
          {
              "_type" : "PublishedMedium",
              "name" : "The New York Times"
          },
          {
              "_type" : "Company",
              "name" : "The New York Times"
          },
          {
              "_type" : "Person",
              "name" : "Eric A. Taub"
          }
      ]
  }
  </code></pre>
  
  <p>I want to do indexing on two fields: 'summary' and 'title' of 'article' array. </p>
  
  <p>So far, what I have learned is putting this entire document into solr does not make sense as it will increase the size of index and make searches slower. So, I decided to store following fields in Solr: 'docId', 'title', 'summary', so while searching in solr I will retrieve only docId and then retrieve other details from mongodb bec it faster than retrieving data from solr bec of analysers, tokenisers and all. </p>
  
  <p><strong>FIRST:</strong>
  So, I need to maintain a unique field 'docId', shall I use use the default '_id' generated by mongod? But for that the doc has to be inserted first, so that mongod can generate the '_id'. So I need to retrieve the doc after inserting into mongodb, fetch the '_id' then insert in solr the fields 'docId', 'summary' and 'title'. Can this be imroved?</p>
  
  <p><strong>SECOND:</strong>
  I need to define the schema in solr for this where I have to map fields from mongodb to fields in solr. I have the default instance of solr running, from solr/example/start.jar.
  It has a default schema and a default collection called 'collection1'. How can I create my own collection, I cannot find anything in the admin interface for this. I want to create a collection for my project and then write a schema as I have defined above.</p>
  
  <p>Whatever tutorials I have found, they simply add the documents to solr.  So do I need to override the default schema?</p>
  
  <p>Note: I am new to Solr, you would have already inferred after reading the question :D So please help!</p>