title: Solr - memory requirement for indexing
body: <p><strong>UPDATE: added some java code and also the command used to start the job</strong></p>
  
  <p>I am running an indexing process as a job on an HPC server which allocates memory to jobs. I hit the memory ceiling during the indexing process, and I would like to ask for my data an usage scenario, what would be minimum memory for 1) indexing and 2) querying the index later.</p>
  
  <p>Scenario:</p>
  
  <ul>
  <li>projected index size to be around 70GB</li>
  <li>projected record # to be around 150 million</li>
  <li>I use Java solr client, solr = 7.1.0</li>
  <li>During indexing I commit every 10k records</li>
  <li>I ran the job with 24GB allocated memory during indexing but it crashed about half way, i.e., 75million records indexed</li>
  <li>Query usage: very light, the index is only to be used by myself (one person) to access the data easily and query will only be sequential and mostly one field based</li>
  </ul>
  
  <p>The Java code below is used to start a server in the 'embedded' mode:</p>
  
  <pre><code>    CoreContainer prodNCContainer = new CoreContainer(args[1]);
      prodNCContainer.load();
      SolrClient prodNameCatDescIndex = new EmbeddedSolrServer(prodNCContainer.getCore("prodcatdesc"));
  </code></pre>
  
  <p>The command I use to start the job is like follows:</p>
  
  <pre><code>#$ -l h_rt=168:00:00 -l rmem=24G -m bea -M ... 
  java -Xmx23000m -cp '........ 1.0-SNAPSHOT-jar-with-dependencies.jar' x.y.z.ProdCatDescIndexCreator ....
  </code></pre>
  
  <p>The first line asks the server to allocate 24G memory to my job.</p>
  
  <p>As I mentioned the job failed because the server killed the job when it exceeds its allocated 24GB memory. This tells me that for this indexing process that is not enough. So what is the rule of thumb to configure the memory during the indexing process? And what about later to use the index?</p>
  
  <p>I remember that you do not need larger memory than the expected index size... that would be unrealistic as well IMO. In the past I manage to load and query a 43G solr index on my laptop that has only max of 16GB memory. I am confused.</p>
  
  <p>Is there any configuration in the solrconfig.xml file to deal with this, e.g., configure number of segments maybe? I am not sure how to do that ...</p>