title: elasticsearch bulk processor failure
body: <p>I want to index a large batch of index requests to 60 different indices(about 1000 million pieces,indices' name are like <code>boss-log-yyyy-MM-dd</code>).</p>
  
  <p>Here is my java code:</p>
  
  <pre><code>List&lt;IndexRequest&gt; indexRequesList = bossMockDataService.indexRequestGenerator(batch); //generate random mock data.
      indexRequesList.forEach(indexRequest -&gt; {
          bulkProcessor.add(indexRequest);
      });
      try {
         return bulkProcessor.awaitClose(30L, TimeUnit.SECONDS);
      } catch (InterruptedException e) {
          e.printStackTrace();
          return false;
      }
  </code></pre>
  
  <p>And here is my bulk processor:</p>
  
  <pre><code>@Override
  public BulkProcessor initESbulkProcessor(RestHighLevelClient client) {
      Settings settings = Settings.EMPTY;
      ThreadPool threadPool = new ThreadPool(settings); // I have no Idea what the 'settings' object is.
      BulkProcessor.Listener listener = new BulkProcessor.Listener() {
          @Override
          public void beforeBulk(long executionId, BulkRequest request) {
              int numberOfActions = request.numberOfActions();
              logger.debug("Executing bulk [{}] with {} requests", executionId, numberOfActions);
          }
  
          @Override
          public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
  
              if (response.hasFailures()) {
                  logger.warn("Bulk [{}] executed with failures", executionId);
              } else {
                  logger.debug("Bulk [{}] completed in {} milliseconds", executionId, response.getTook().getMillis());
              }
          }
  
          @Override
          public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
  
              logger.error("Failed to execute bulk", failure);
          }
  
      };
      BulkProcessor.Builder builder = new BulkProcessor.Builder(client::bulkAsync, listener, threadPool);
      BulkProcessor bulkProcessor = builder.build();
      builder.setBulkActions(2000); 
      builder.setBulkSize(new ByteSizeValue(10L, ByteSizeUnit.MB));
      builder.setConcurrentRequests(10);
      builder.setFlushInterval(TimeValue.timeValueSeconds(10L));
      builder.setBackoffPolicy(BackoffPolicy.constantBackoff(TimeValue.timeValueSeconds(1L), 3));
      logger.info("bulk processor build complete!");
      return bulkProcessor;
  
  }
  </code></pre>
  
  <p>It seems some data index request failed.The log is like:</p>
  
  <pre><code>2017-11-15 16:42:24 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [96] executed with failures
  2017-11-15 16:42:24 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [97] executed with failures
  2017-11-15 16:42:24 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [98] executed with failures
  2017-11-15 16:42:24 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [99] executed with failures
  2017-11-15 16:42:24 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [100] executed with failures
  2017-11-15 16:42:24 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [101] executed with failures
  2017-11-15 16:42:24 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [102] executed with failures
  2017-11-15 16:42:24 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [103] executed with failures
  2017-11-15 16:42:25 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [104] executed with failures
  2017-11-15 16:42:25 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [105] executed with failures
  2017-11-15 16:42:25 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [106] executed with failures
  2017-11-15 16:42:25 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [107] executed with failures
  2017-11-15 16:42:25 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [108] executed with failures
  2017-11-15 16:42:25 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [109] executed with failures
  2017-11-15 16:42:25 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [110] executed with failures
  2017-11-15 16:42:26 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [111] executed with failures
  2017-11-15 16:42:26 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [112] executed with failures
  2017-11-15 16:42:26 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [113] executed with failures
  2017-11-15 16:42:26 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [114] executed with failures
  2017-11-15 16:42:26 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [115] executed with failures
  2017-11-15 16:42:26 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [116] executed with failures
  2017-11-15 16:42:26 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [117] executed with failures
  2017-11-15 16:42:27 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [118] executed with failures
  2017-11-15 16:42:27 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [119] executed with failures
  2017-11-15 16:42:27 [I/O dispatcher 1] WARN  BossMockDataServiceImpl:220 - Bulk [120] executed with failures
  </code></pre>
  
  <p>My situation:</p>
  
  <ul>
  <li>New to ES.</li>
  <li>1 million pieces of data per day need to be stored and queried.</li>
  <li>Yet I have not configured my shard/replica/node/cluster,they are default now.</li>
  <li>When I use MySQL,I create a new partition for a day's data.</li>
  <li>The limit for MySQL is 60 days.The IO operation is getting more and more slow.</li>
  </ul>
  
  <p>My goal:</p>
  
  <ul>
  <li>Easy to maintain my ES.</li>
  <li>Fast query,update and aggregation.</li>
  <li>Store 60+ days data. </li>
  <li>No tolerance for bulk request failure.(Seems some data lost above)</li>
  </ul>
  
  <p>What I wonder is :</p>
  
  <ul>
  <li><p>Is it a good idea to create new index for a day's data ? Or I put them in one index?</p></li>
  <li><p>Why the bulk processor call back is failure ? How to fix it ?</p></li>
  <li><p>Is it necessary to create more cluster/node ? Where should I begin?</p></li>
  </ul>