title: Hibernate Search: concurrent timeout exception
body: <p><strong>Scenario:</strong>
  I am processing an excel file containing 150+columns and 15000 rows. This file converted to List and then called repository.saveAll(List) . This service method was called concurently by 15 user request.</p>
  <p><strong>Code</strong></p>
  <pre><code>@Transactional(rollbackFor = Exception.class)
      public BooleanResponse fileProcessing(
              Request req) {
  
  
          for (MultipartFile file : req.getMasterSheets()) {
              List&lt;EntityClazz&gt; list = excelParser.processExcelFile(file, EntityClazz.class);
              EntityClazzRepository.saveAll(list);
              list.clear();
          }
  
          return BooleanResponse.success();
  
      }
  </code></pre>
  <p>I am facing following exception with Hibernate Search and Elastic Search.</p>
  <pre><code>2019-11-14 12:38:32.567 ERROR 1 --- [eout executor-1] o.h.s.exception.impl.LogErrorHandler     : HSEARCH000058: Exception occurred org.hibernate.search.exception.SearchException: HSEARCH400007: Elasticsearch request failed.
  Request: POST /_bulk with parameters {refresh=false}
  Response: null
  Subsequent failures:
          Entity com.xyz.xyz.domain.entity.epc.EntityClazz  Id 436294  Work Type  org.hibernate.search.backend.AddLuceneWork
          Entity com.xyz.xyz.domain.entity.epc.EntityClazz  Id 436310  Work Type  org.hibernate.search.backend.AddLuceneWork
          Entity com.xyz.xyz.domain.entity.epc.EntityClazz  Id 436326  Work Type  org.hibernate.search.backend.AddLuceneWork
          Entity com.xyz.xyz.domain.entity.epc.EntityClazz  Id 436342  Work Type  org.hibernate.search.backend.AddLuceneWork
  .
  .
  .
  .
  
          Entity com.xyz.xyz.domain.entity.epc.EntityClazz  Id 472620  Work Type  org.hibernate.search.backend.AddLuceneWork
          Entity com.xyz.xyz.domain.entity.epc.EntityClazz  Id 472625  Work Type  org.hibernate.search.backend.AddLuceneWork
  
  
  
  
  org.hibernate.search.exception.SearchException: HSEARCH400007: Elasticsearch request failed.
  Request: POST /_bulk with parameters {refresh=false}
  Response: null
          at org.hibernate.search.elasticsearch.work.impl.BulkWork.lambda$execute$1(BulkWork.java:77) ~[hibernate-search-elasticsearch-5.10.5.Final.jar!/:5.10.5.Final]
          at org.hibernate.search.util.impl.Futures.lambda$handler$1(Futures.java:57) ~[hibernate-search-engine-5.10.5.Final.jar!/:5.10.5.Final]
          at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870) ~[na:1.8.0_212]
          at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852) ~[na:1.8.0_212]
          at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_212]
          at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[na:1.8.0_212]
          at org.hibernate.search.elasticsearch.client.impl.DefaultElasticsearchClient.lambda$send$3(DefaultElasticsearchClient.java:137) ~[hibernate-search-elasticsearch-5.10.5.Final.jar!/:5.10.5.Final]
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_212]
          at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_212]
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_212]
          at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_212]
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_212]
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_212]
          at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_212]
  Caused by: java.util.concurrent.TimeoutException: null
          ... 8 common frames omitted
  </code></pre>
  <p>This log is one such example of many exceptions printed out... Feels like there are many threads working and some how failed. Don't know why.. Data set which needs to be indexed was almost 50000 at one time. and Hibernate search starts indexing automatically and asynchronously after transaction commit as I have set async true.</p>
  <p>I am using Hibernate Search version 5.10 and elastic search version 5.6.10.
  My database table is very large. and There is bulk update/insert operation is performed by Hibernate on this table.</p>
  <p>My Elastic Search Server is on default configuration. that only one node. yellow health.</p>
  <p>Here is Elastic Search Cluster Node stats:</p>
  <pre><code>{
     &quot;_nodes&quot;:{
        &quot;total&quot;:1,
        &quot;successful&quot;:1,
        &quot;failed&quot;:0
     },
     &quot;cluster_name&quot;:&quot;elasticsearch&quot;,
     &quot;nodes&quot;:{
        &quot;I7fWwFhORhimHRDL6jJTuw&quot;:{
           &quot;timestamp&quot;:1573740483870,
           &quot;name&quot;:&quot;I7fWwFh&quot;,
           &quot;transport_address&quot;:&quot;&quot;,
           &quot;host&quot;:&quot;&quot;,
           &quot;ip&quot;:&quot;&quot;,
           &quot;roles&quot;:[
              &quot;master&quot;,
              &quot;data&quot;,
              &quot;ingest&quot;
           ],
           &quot;indices&quot;:{
              &quot;docs&quot;:{
                 &quot;count&quot;:7090,
                 &quot;deleted&quot;:956
              },
              &quot;store&quot;:{
                 &quot;size_in_bytes&quot;:69228643,
                 &quot;throttle_time_in_millis&quot;:0
              },
              &quot;indexing&quot;:{
                 &quot;index_total&quot;:5091795,
                 &quot;index_time_in_millis&quot;:60401979,
                 &quot;index_current&quot;:11,
                 &quot;index_failed&quot;:7,
                 &quot;delete_total&quot;:124964,
                 &quot;delete_time_in_millis&quot;:2232,
                 &quot;delete_current&quot;:0,
                 &quot;noop_update_total&quot;:0,
                 &quot;is_throttled&quot;:true,
                 &quot;throttle_time_in_millis&quot;:1348474
              },
              &quot;get&quot;:{
                 &quot;total&quot;:4913,
                 &quot;time_in_millis&quot;:316,
                 &quot;exists_total&quot;:4913,
                 &quot;exists_time_in_millis&quot;:316,
                 &quot;missing_total&quot;:0,
                 &quot;missing_time_in_millis&quot;:0,
                 &quot;current&quot;:0
              },
              &quot;search&quot;:{
                 &quot;open_contexts&quot;:0,
                 &quot;query_total&quot;:232976,
                 &quot;query_time_in_millis&quot;:79853,
                 &quot;query_current&quot;:0,
                 &quot;fetch_total&quot;:183622,
                 &quot;fetch_time_in_millis&quot;:608420,
                 &quot;fetch_current&quot;:0,
                 &quot;scroll_total&quot;:3555,
                 &quot;scroll_time_in_millis&quot;:192717,
                 &quot;scroll_current&quot;:0,
                 &quot;suggest_total&quot;:0,
                 &quot;suggest_time_in_millis&quot;:0,
                 &quot;suggest_current&quot;:0
              },
              &quot;merges&quot;:{
                 &quot;current&quot;:0,
                 &quot;current_docs&quot;:0,
                 &quot;current_size_in_bytes&quot;:0,
                 &quot;total&quot;:9867,
                 &quot;total_time_in_millis&quot;:23061388,
                 &quot;total_docs&quot;:7412149,
                 &quot;total_size_in_bytes&quot;:191974966202,
                 &quot;total_stopped_time_in_millis&quot;:0,
                 &quot;total_throttled_time_in_millis&quot;:626635,
                 &quot;total_auto_throttle_in_bytes&quot;:80244256738
              },
              &quot;refresh&quot;:{
                 &quot;total&quot;:200577,
                 &quot;total_time_in_millis&quot;:32611981,
                 &quot;listeners&quot;:0
              },
              &quot;flush&quot;:{
                 &quot;total&quot;:7747,
                 &quot;total_time_in_millis&quot;:67635
              },
              &quot;warmer&quot;:{
                 &quot;current&quot;:0,
                 &quot;total&quot;:677,
                 &quot;total_time_in_millis&quot;:21
              },
              &quot;query_cache&quot;:{
                 &quot;memory_size_in_bytes&quot;:0,
                 &quot;total_count&quot;:0,
                 &quot;hit_count&quot;:0,
                 &quot;miss_count&quot;:0,
                 &quot;cache_size&quot;:0,
                 &quot;cache_count&quot;:0,
                 &quot;evictions&quot;:0
              },
              &quot;fielddata&quot;:{
                 &quot;memory_size_in_bytes&quot;:0,
                 &quot;evictions&quot;:0
              },
              &quot;completion&quot;:{
                 &quot;size_in_bytes&quot;:0
              },
              &quot;segments&quot;:{
                 &quot;count&quot;:48,
                 &quot;memory_in_bytes&quot;:4551470,
                 &quot;terms_memory_in_bytes&quot;:3595446,
                 &quot;stored_fields_memory_in_bytes&quot;:20168,
                 &quot;term_vectors_memory_in_bytes&quot;:0,
                 &quot;norms_memory_in_bytes&quot;:913856,
                 &quot;points_memory_in_bytes&quot;:280,
                 &quot;doc_values_memory_in_bytes&quot;:21720,
                 &quot;index_writer_memory_in_bytes&quot;:0,
                 &quot;version_map_memory_in_bytes&quot;:0,
                 &quot;fixed_bit_set_memory_in_bytes&quot;:0,
                 &quot;max_unsafe_auto_id_timestamp&quot;:-1,
                 &quot;file_sizes&quot;:{
  
                 }
              },
              &quot;translog&quot;:{
                 &quot;operations&quot;:0,
                 &quot;size_in_bytes&quot;:1290
              },
              &quot;request_cache&quot;:{
                 &quot;memory_size_in_bytes&quot;:21920,
                 &quot;evictions&quot;:0,
                 &quot;hit_count&quot;:0,
                 &quot;miss_count&quot;:30
              },
              &quot;recovery&quot;:{
                 &quot;current_as_source&quot;:0,
                 &quot;current_as_target&quot;:0,
                 &quot;throttle_time_in_millis&quot;:0
              }
           },
           &quot;os&quot;:{
              &quot;timestamp&quot;:1573740483888,
              &quot;cpu&quot;:{
                 &quot;percent&quot;:0,
                 &quot;load_average&quot;:{
                    &quot;1m&quot;:0,
                    &quot;5m&quot;:0,
                    &quot;15m&quot;:0
                 }
              },
              &quot;mem&quot;:{
                 &quot;total_in_bytes&quot;:4134809600,
                 &quot;free_in_bytes&quot;:115888128,
                 &quot;used_in_bytes&quot;:4018921472,
                 &quot;free_percent&quot;:3,
                 &quot;used_percent&quot;:97
              },
              &quot;swap&quot;:{
                 &quot;total_in_bytes&quot;:0,
                 &quot;free_in_bytes&quot;:0,
                 &quot;used_in_bytes&quot;:0
              },
              &quot;cgroup&quot;:{
                 &quot;cpuacct&quot;:{
                    &quot;control_group&quot;:&quot;/&quot;,
                    &quot;usage_nanos&quot;:85077837036370
                 },
                 &quot;cpu&quot;:{
                    &quot;control_group&quot;:&quot;/&quot;,
                    &quot;cfs_period_micros&quot;:100000,
                    &quot;cfs_quota_micros&quot;:-1,
                    &quot;stat&quot;:{
                       &quot;number_of_elapsed_periods&quot;:0,
                       &quot;number_of_times_throttled&quot;:0,
                       &quot;time_throttled_nanos&quot;:0
                    }
                 }
              }
           },
           &quot;process&quot;:{
              &quot;timestamp&quot;:1573740483888,
              &quot;open_file_descriptors&quot;:213,
              &quot;max_file_descriptors&quot;:65536,
              &quot;cpu&quot;:{
                 &quot;percent&quot;:0,
                 &quot;total_in_millis&quot;:82774150
              },
              &quot;mem&quot;:{
                 &quot;total_virtual_in_bytes&quot;:5024968704
              }
           },
           &quot;jvm&quot;:{
              &quot;timestamp&quot;:1573740483888,
              &quot;uptime_in_millis&quot;:2672036639,
              &quot;mem&quot;:{
                 &quot;heap_used_in_bytes&quot;:1155032416,
                 &quot;heap_used_percent&quot;:54,
                 &quot;heap_committed_in_bytes&quot;:2130051072,
                 &quot;heap_max_in_bytes&quot;:2130051072,
                 &quot;non_heap_used_in_bytes&quot;:146981608,
                 &quot;non_heap_committed_in_bytes&quot;:154607616,
                 &quot;pools&quot;:{
                    &quot;young&quot;:{
                       &quot;used_in_bytes&quot;:135568192,
                       &quot;max_in_bytes&quot;:139591680,
                       &quot;peak_used_in_bytes&quot;:139591680,
                       &quot;peak_max_in_bytes&quot;:139591680
                    },
                    &quot;survivor&quot;:{
                       &quot;used_in_bytes&quot;:4866896,
                       &quot;max_in_bytes&quot;:17432576,
                       &quot;peak_used_in_bytes&quot;:17432576,
                       &quot;peak_max_in_bytes&quot;:17432576
                    },
                    &quot;old&quot;:{
                       &quot;used_in_bytes&quot;:1014597328,
                       &quot;max_in_bytes&quot;:1973026816,
                       &quot;peak_used_in_bytes&quot;:1638372912,
                       &quot;peak_max_in_bytes&quot;:1973026816
                    }
                 }
              },
              &quot;threads&quot;:{
                 &quot;count&quot;:43,
                 &quot;peak_count&quot;:50
              },
              &quot;gc&quot;:{
                 &quot;collectors&quot;:{
                    &quot;young&quot;:{
                       &quot;collection_count&quot;:170449,
                       &quot;collection_time_in_millis&quot;:1934820
                    },
                    &quot;old&quot;:{
                       &quot;collection_count&quot;:847,
                       &quot;collection_time_in_millis&quot;:38599
                    }
                 }
              },
              &quot;buffer_pools&quot;:{
                 &quot;direct&quot;:{
                    &quot;count&quot;:32,
                    &quot;used_in_bytes&quot;:67767443,
                    &quot;total_capacity_in_bytes&quot;:67767442
                 },
                 &quot;mapped&quot;:{
                    &quot;count&quot;:108,
                    &quot;used_in_bytes&quot;:68478134,
                    &quot;total_capacity_in_bytes&quot;:68478134
                 }
              },
              &quot;classes&quot;:{
                 &quot;current_loaded_count&quot;:12473,
                 &quot;total_loaded_count&quot;:12527,
                 &quot;total_unloaded_count&quot;:54
              }
           },
           &quot;thread_pool&quot;:{
              &quot;bulk&quot;:{
                 &quot;threads&quot;:2,
                 &quot;queue&quot;:0,
                 &quot;active&quot;:0,
                 &quot;rejected&quot;:0,
                 &quot;largest&quot;:2,
                 &quot;completed&quot;:383422
              },
              &quot;fetch_shard_started&quot;:{
                 &quot;threads&quot;:1,
                 &quot;queue&quot;:0,
                 &quot;active&quot;:0,
                 &quot;rejected&quot;:0,
                 &quot;largest&quot;:4,
                 &quot;completed&quot;:97
              },
              &quot;fetch_shard_store&quot;:{
                 &quot;threads&quot;:0,
                 &quot;queue&quot;:0,
                 &quot;active&quot;:0,
                 &quot;rejected&quot;:0,
                 &quot;largest&quot;:0,
                 &quot;completed&quot;:0
              },
              &quot;flush&quot;:{
                 &quot;threads&quot;:1,
                 &quot;queue&quot;:0,
                 &quot;active&quot;:0,
                 &quot;rejected&quot;:0,
                 &quot;largest&quot;:1,
                 &quot;completed&quot;:12854
              },
              &quot;force_merge&quot;:{
                 &quot;threads&quot;:1,
                 &quot;queue&quot;:0,
                 &quot;active&quot;:0,
                 &quot;rejected&quot;:0,
                 &quot;largest&quot;:1,
                 &quot;completed&quot;:1240
              },
              &quot;generic&quot;:{
                 &quot;threads&quot;:4,
                 &quot;queue&quot;:0,
                 &quot;active&quot;:0,
                 &quot;rejected&quot;:0,
                 &quot;largest&quot;:8,
                 &quot;completed&quot;:273461
              },
              &quot;get&quot;:{
                 &quot;threads&quot;:2,
                 &quot;queue&quot;:0,
                 &quot;active&quot;:0,
                 &quot;rejected&quot;:0,
                 &quot;largest&quot;:2,
                 &quot;completed&quot;:4897
              },
              &quot;index&quot;:{
                 &quot;threads&quot;:2,
                 &quot;queue&quot;:0,
                 &quot;active&quot;:0,
                 &quot;rejected&quot;:0,
                 &quot;largest&quot;:2,
                 &quot;completed&quot;:16
              },
              &quot;listener&quot;:{
                 &quot;threads&quot;:1,
                 &quot;queue&quot;:0,
                 &quot;active&quot;:0,
                 &quot;rejected&quot;:0,
                 &quot;largest&quot;:1,
                 &quot;completed&quot;:21
              },
              &quot;management&quot;:{
                 &quot;threads&quot;:3,
                 &quot;queue&quot;:0,
                 &quot;active&quot;:1,
                 &quot;rejected&quot;:0,
                 &quot;largest&quot;:3,
                 &quot;completed&quot;:366004
              },
              &quot;refresh&quot;:{
                 &quot;threads&quot;:1,
                 &quot;queue&quot;:0,
                 &quot;active&quot;:0,
                 &quot;rejected&quot;:0,
                 &quot;largest&quot;:1,
                 &quot;completed&quot;:21459450
              },
              &quot;search&quot;:{
                 &quot;threads&quot;:4,
                 &quot;queue&quot;:0,
                 &quot;active&quot;:0,
                 &quot;rejected&quot;:0,
                 &quot;largest&quot;:4,
                 &quot;completed&quot;:463814
              },
              &quot;snapshot&quot;:{
                 &quot;threads&quot;:0,
                 &quot;queue&quot;:0,
                 &quot;active&quot;:0,
                 &quot;rejected&quot;:0,
                 &quot;largest&quot;:0,
                 &quot;completed&quot;:0
              },
              &quot;warmer&quot;:{
                 &quot;threads&quot;:0,
                 &quot;queue&quot;:0,
                 &quot;active&quot;:0,
                 &quot;rejected&quot;:0,
                 &quot;largest&quot;:0,
                 &quot;completed&quot;:0
              }
           },
           &quot;fs&quot;:{
              &quot;timestamp&quot;:1573740483888,
              &quot;total&quot;:{
                 &quot;total_in_bytes&quot;:8577331200,
                 &quot;free_in_bytes&quot;:20480,
                 &quot;available_in_bytes&quot;:20480
              },
              &quot;data&quot;:[
                 {
                    &quot;path&quot;:&quot;/var/lib/elasticsearch/nodes/0&quot;,
                    &quot;mount&quot;:&quot;/ (/dev/xvda1)&quot;,
                    &quot;type&quot;:&quot;xfs&quot;,
                    &quot;total_in_bytes&quot;:8577331200,
                    &quot;free_in_bytes&quot;:20480,
                    &quot;available_in_bytes&quot;:20480,
                    &quot;spins&quot;:&quot;false&quot;
                 }
              ],
              &quot;io_stats&quot;:{
                 &quot;devices&quot;:[
                    {
                       &quot;device_name&quot;:&quot;xvda1&quot;,
                       &quot;operations&quot;:5857367,
                       &quot;read_operations&quot;:171280,
                       &quot;write_operations&quot;:5686087,
                       &quot;read_kilobytes&quot;:13911088,
                       &quot;write_kilobytes&quot;:304080912
                    }
                 ],
                 &quot;total&quot;:{
                    &quot;operations&quot;:5857367,
                    &quot;read_operations&quot;:171280,
                    &quot;write_operations&quot;:5686087,
                    &quot;read_kilobytes&quot;:13911088,
                    &quot;write_kilobytes&quot;:304080912
                 }
              }
           },
           &quot;transport&quot;:{
              &quot;server_open&quot;:0,
              &quot;rx_count&quot;:20,
              &quot;rx_size_in_bytes&quot;:7538,
              &quot;tx_count&quot;:20,
              &quot;tx_size_in_bytes&quot;:7538
           },
           &quot;http&quot;:{
              &quot;current_open&quot;:20,
              &quot;total_opened&quot;:2915
           },
           &quot;breakers&quot;:{
              &quot;request&quot;:{
                 &quot;limit_size_in_bytes&quot;:1278030643,
                 &quot;limit_size&quot;:&quot;1.1gb&quot;,
                 &quot;estimated_size_in_bytes&quot;:0,
                 &quot;estimated_size&quot;:&quot;0b&quot;,
                 &quot;overhead&quot;:1,
                 &quot;tripped&quot;:0
              },
              &quot;fielddata&quot;:{
                 &quot;limit_size_in_bytes&quot;:1278030643,
                 &quot;limit_size&quot;:&quot;1.1gb&quot;,
                 &quot;estimated_size_in_bytes&quot;:0,
                 &quot;estimated_size&quot;:&quot;0b&quot;,
                 &quot;overhead&quot;:1.03,
                 &quot;tripped&quot;:0
              },
              &quot;in_flight_requests&quot;:{
                 &quot;limit_size_in_bytes&quot;:2130051072,
                 &quot;limit_size&quot;:&quot;1.9gb&quot;,
                 &quot;estimated_size_in_bytes&quot;:4396991,
                 &quot;estimated_size&quot;:&quot;4.1mb&quot;,
                 &quot;overhead&quot;:1,
                 &quot;tripped&quot;:0
              },
              &quot;parent&quot;:{
                 &quot;limit_size_in_bytes&quot;:1491035750,
                 &quot;limit_size&quot;:&quot;1.3gb&quot;,
                 &quot;estimated_size_in_bytes&quot;:4396991,
                 &quot;estimated_size&quot;:&quot;4.1mb&quot;,
                 &quot;overhead&quot;:1,
                 &quot;tripped&quot;:0
              }
           },
           &quot;script&quot;:{
              &quot;compilations&quot;:1,
              &quot;cache_evictions&quot;:0
           },
           &quot;discovery&quot;:{
              &quot;cluster_state_queue&quot;:{
                 &quot;total&quot;:0,
                 &quot;pending&quot;:0,
                 &quot;committed&quot;:0
              }
           },
           &quot;ingest&quot;:{
              &quot;total&quot;:{
                 &quot;count&quot;:0,
                 &quot;time_in_millis&quot;:0,
                 &quot;current&quot;:0,
                 &quot;failed&quot;:0
              },
              &quot;pipelines&quot;:{
  
              }
           }
        }
     }
  }
  </code></pre>
  <p>Here is my Hibernate Search configurations with Spring boot 2.0 application.</p>
  <pre><code>spring:
  
    datasource:
      
      hikari:
        maximum-pool-size: 30
        minimum-idle: 10
    jpa:
      show-sql: false
      properties:
        hibernate:
          ejb:
            interceptor: com.xyz.xyz.xyz.util.EntityInterceptor
          jdbc:
            batch_size: 100
            order_inserts: true
            order_update: true
          search:
            autoregister_listeners: true
            fulltext_query: ERROR
            default_null_token: N/A
            error_handler: com.xyz.xyz.xyz.search.exceptions.HibernateSearchErrorHandler
            default:
              indexmanager: elasticsearch
              worker:
                execution: async
              elasticsearch:
                required_index_status: yellow
                index_schema_management_strategy: none
                host: ${ES_HOST:http://127.0.0.1:9200}
                request: ERROR
                log:
                  json_pretty_printing: false
  </code></pre>
  <p>I am not understanding what is missing. Please guide me in right direction. Ask me if any other information require to understand my problem. Many thanks in advance.</p>