title: Sonarqube upgrade to 6.7: Unrecoverable indexation failures
body: <p>Currently we are in middle of upgrade process from v4.5.7 to v6.7.1.</p>
  
  <p>SonarQube is successfully upgraded to v5.6.7 (including DB upgrade) and currently in the process of updating to v6.7.1. </p>
  
  <p>New SonarQube version started on an empty DB (<a href="http://www.h2database.com/" rel="noreferrer">bundled H2 DB</a>) worked fine. After which all the plugins and their versions are validated and updated accordingly. </p>
  
  <p>Now Sonar service launching is failing with error <code>Background initialization failed. Stopping SonarQube java.lang.IllegalStateException: Unrecoverable indexation failures</code>.</p>
  
  <p>I removed <code>data/es5</code> folder and reattempted to start <code>sonar</code> but with no luck. Any assistance to resolve the issue is greatly appreciated.</p>
  
  <p>Web log:</p>
  
  <pre><code>  INFO  web[][o.s.p.ProcessEntryPoint] Starting web
    INFO  web[][o.a.t.u.n.NioSelectorPool] Using a shared selector for servlet write/read
    INFO  web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin]
    INFO  web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.join.ParentJoinPlugin]
    INFO  web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
    INFO  web[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
    INFO  web[][o.s.s.e.EsClientProvider] Connected to local Elasticsearch: [127.0.0.1:9002]
    INFO  web[][o.s.s.p.LogServerVersion] SonarQube Server / 6.7.1.35068 / 426519346f51f7b980a76f9050f983110550509d
    INFO  web[][o.sonar.db.Database] Create JDBC data source for jdbc:oracle:thin:@host:****:instance
    INFO  web[][o.s.s.p.ServerFileSystemImpl] SonarQube home: /.../sonar/sonarqube-6.7.1
    INFO  web[][o.s.s.u.SystemPasscodeImpl] System authentication by passcode is disabled
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Branch / 1.0.0.507 / e9d161eaa2993f79890dd5d08230f3f566e6be93
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Checkmarx / 8.41.0-20161229-0945
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Developer / 1.0.0.240 / 25c709aeb128caa14725b943422cca19def50493
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Flex / 2.3 / 79167f762219bea1fd181ff5ef6ca4b34abd1675
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin GitHub / 1.4.2.1027 / dcfb02a5fa40c219d527563f5b8c793966523ca7
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Governance / 2.0.2.3011 / 4756b2f9d98d28cf2762d91eba1b48839a1edbe1
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Groovy / 1.5 / dcbcd1eeb8f61c8351e5dd1e51e11e52d186df35
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin LDAP / 2.2.0.608 / 79dc3fa4393a29667673c70182f3016288b548b7
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin License / 3.3.0.1341 / 6b63240af64c92afe9421ece35c99c70f1b97991
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Pitest / 0.9 / 0c011514ce025e5524fdcaa0c917eacf9d40f312
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SVG Badges / 3.0.1
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarABAP / 3.5.0.1080 / c37baa20eb9d24a27961ffe25d95ae4b716a3193
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarC# / 6.5.0.3766 / 6011a1e9fb002edb044ae3092d92bcfb7d9d8a2d
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarCFamily / 5.0.0.9359 / 6001ee3bce59e4bb9a17e5edbe40b4b99bc2901a
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarCOBOL / 4.1.1.2663 / 846e542a08e9e472f0216da6c30e66a96bcbc821
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarJS / 3.2.0.5506 / 86da09565f891121df248835665f5d8ed98772b6
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarJava / 4.15.0.12310 / 572454b93016ec73a53fe0e07b2ffdc356d21ba9
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPHP / 2.11.0.2485 / 741861a29e5f9a26c6c99c06268facb6c4f4a882
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPLI / 1.7.0.1117 / 2e07afcb9ecb6a2e96c980b48f7195b82ffdd014
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPLSQL / 3.2.0.1753 / 455f471cfde991b2cd1a86a939ff410bf3006059
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarPython / 1.8.0.1496 / 3fe3bc4d0273a5721ea2fb368dc45b1bb82fede3
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarQube :: Plugins :: SCM :: Git / 1.3.0.869 / 4da53e3f9e55f4f2e5796625cb0c5768ed152079
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarQube :: Plugins :: SCM :: SVN / 1.6.0.860 / 2111fdbd1dddda4ad6d4ed6486fd0b18c1010d3b
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarRPG / 2.2.0.1005 / 146ae32dade755799da4fa9655e865d8757fddfa
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarSwift / 3.1.0.2067 / 22db8593d10c835675860598372d41ba8d5f0b8e
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarTS / 1.1.0.1079 / 042c9e65239a47d92d305f9767f730b3cc1e5ed3
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarTSQL / 1.1.0.2177 / e8811dfecb71d328c4deca89297bc100b270313c
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarVB / 4.2.0.248 / 75e2175c7464b814699abd6719b964c027717683
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarVB6 / 2.3.0.992 / 8c9e2016cb2e325b2fdeba4d303181b33bbbda4b
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin SonarXML / 1.4.3.1027 / 39588245cecf538bb27be4e496ff303b0143d20b
    INFO  web[][o.s.s.p.ServerPluginRepository] Deploy plugin Web / 2.5.0.476 / 636872f5d37fa7a440fe07d08d504e1a881225e5
    INFO  web[][c.c.sonar.CxPlugin] Checkmarx plugin loaded
    INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceFilter@6cdba960 [pattern=UrlPattern{inclusions=[/api/system/migrate_db/*, ...], exclusions=[/api/properties*, ...]}]
    INFO  web[][o.s.s.a.EmbeddedTomcat] HTTP connector enabled on port ****
    INFO  web[][o.s.s.p.UpdateCenterClient] Update center: https://update.sonarsource.org/update-center.properties (no proxy)
    INFO  web[][c.q.p.s.b.f.PreferredFontProvider] SVGImageGenerator will be using font 'Verdana' in order to compute SVG badges width.
    INFO  web[][c.q.p.s.b.w.SVGImageGenerator] SVGImageGenerator is now ready.
    INFO  web[][c.q.p.s.b.w.g.QualityGateBadgeGenerator] QualityGateBadgeGenerator is now ready.
    INFO  web[][c.q.p.s.b.w.m.MeasureBadgeGenerator] MeasureBadgeGenerator is now ready.
    INFO  web[][c.q.p.s.b.w.c.CeActivityBadgeGenerator] CeActivityBadgeGenerator is now ready.
    WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/editions/show_license
    WARN  web[][o.s.a.s.w.WebService$Action] The response example is not set on action api/views/definition
    INFO  web[][o.s.s.s.LogServerId] Server ID: *********
    WARN  web[][o.s.s.u.DeprecatedViews] Widget 'Checkmarx' (checkmarx-project-widget) is ignored. See org.sonar.api.web.page.PageDefinition to define pages.
    WARN  web[][o.s.s.u.DeprecatedViews] Page 'Checkmarx' (/cx_config/index) is ignored. See org.sonar.api.web.page.PageDefinition to define pages.
    INFO  web[][org.sonar.INFO] Security realm: LDAP
    INFO  web[][org.sonar.INFO] Security realm started
    INFO  web[][o.s.s.t.TelemetryDaemon] Sharing of SonarQube statistics is enabled.
    INFO  web[][o.s.s.n.NotificationDaemon] Notification service started (delay 60 sec.)
    INFO  web[][o.s.s.s.GeneratePluginIndex] Generate scanner plugin index
    INFO  web[][o.s.s.s.RegisterPlugins] Register plugins
    INFO  web[][o.s.s.s.RegisterMetrics] Register metrics
    INFO  web[][o.s.s.r.RegisterRules] Register rules
    INFO  web[][o.s.s.q.BuiltInQProfileRepositoryImpl] Load quality profiles
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Register quality profiles
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile cpp/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile c/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile rpg/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile js/Sonar way Recommended
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile js/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile py/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile pli/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile tsql/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile vb/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile plsql/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile cs/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile java/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile cobol/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile web/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile flex/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile xml/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile php/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile php/PSR-2
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile php/Drupal
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile vbnet/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile abap/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile grvy/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile objc/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile swift/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile ts/Sonar way
    INFO  web[][o.s.s.q.RegisterQualityProfiles] Update profile ts/Sonar way recommended
    INFO  web[][o.s.s.s.RegisterPermissionTemplates] Register permission templates
    INFO  web[][o.s.s.s.RenameDeprecatedPropertyKeys] Rename deprecated property keys
    INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceFilter@1c655890 [pattern=UrlPattern{inclusions=[/api/measures/component/*, ...], exclusions=[/api/properties*, ...]}]
    INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.DeprecatedPropertiesWsFilter@4f45e5bd [pattern=UrlPattern{inclusions=[/api/properties/*], exclusions=[]}]
    INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.ws.WebServiceReroutingFilter@29719313 [pattern=UrlPattern{inclusions=[/api/components/bulk_update_key, ...], exclusions=[]}]
    INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.InitFilter@694be25 [pattern=UrlPattern{inclusions=[/sessions/init/*], exclusions=[]}]
    INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.OAuth2CallbackFilter@61014c56 [pattern=UrlPattern{inclusions=[/oauth2/callback/*], exclusions=[]}]
    INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.LoginAction@44a4512f [pattern=UrlPattern{inclusions=[/api/authentication/login], exclusions=[]}]
    INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.LogoutAction@67e2e76 [pattern=UrlPattern{inclusions=[/api/authentication/logout], exclusions=[]}]
    INFO  web[][o.s.s.p.w.MasterServletFilter] Initializing servlet filter org.sonar.server.authentication.ws.ValidateAction@12d61b62 [pattern=UrlPattern{inclusions=[/api/authentication/validate], exclusions=[]}]
    INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [components/component] ...
    INFO  web[][o.s.s.es.BulkIndexer] 201463 requests processed (3357 items/sec)
    INFO  web[][o.s.s.es.BulkIndexer] 357701 requests processed (2603 items/sec)
    INFO  web[][o.s.s.es.BulkIndexer] 506390 requests processed (2478 items/sec)
    INFO  web[][o.s.s.es.BulkIndexer] 636193 requests processed (2163 items/sec)
    INFO  web[][o.s.s.es.BulkIndexer] 766613 requests processed (2173 items/sec)
    INFO  web[][o.s.s.es.BulkIndexer] 899827 requests processed (2220 items/sec)
    INFO  web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec)
    INFO  web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec)
    INFO  web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec)
    INFO  web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec)
    INFO  web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec)
    INFO  web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec)
    INFO  web[][o.s.s.es.BulkIndexer] 899827 requests processed (0 items/sec)
    INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [components/component] done | time=815372ms
    INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [views/view] ...
    INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [views/view] done | time=3122ms
    INFO  web[][o.s.s.e.IndexerStartupTask] Indexing of type [issues/issue] ...
    INFO  web[][o.s.s.es.BulkIndexer] 758897 requests processed (12648 items/sec)
    INFO  web[][o.s.s.es.BulkIndexer] 1395232 requests processed (10605 items/sec)
    INFO  web[][o.s.s.es.BulkIndexer] 1989212 requests processed (9899 items/sec)
    INFO  web[][o.s.s.es.BulkIndexer] 2567812 requests processed (9643 items/sec)
    INFO  web[][o.s.s.es.BulkIndexer] 3159399 requests processed (9859 items/sec)
    INFO  web[][o.s.s.es.BulkIndexer] 3320485 requests processed (2684 items/sec)
    INFO  web[][o.s.s.es.BulkIndexer] 3320485 requests processed (0 items/sec)
    ERROR web[][o.s.s.p.Platform] Background initialization failed. Stopping SonarQube
    java.lang.IllegalStateException: Unrecoverable indexation failures
  at org.sonar.server.es.IndexingListener$1.onFinish(IndexingListener.java:39)
  at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:117)
  at org.sonar.server.issue.index.IssueIndexer.doIndex(IssueIndexer.java:247)
  at org.sonar.server.issue.index.IssueIndexer.indexOnStartup(IssueIndexer.java:95)
  at org.sonar.server.es.IndexerStartupTask.indexUninitializedTypes(IndexerStartupTask.java:68)
  at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
  at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
  at org.sonar.server.es.IndexerStartupTask.execute(IndexerStartupTask.java:55)
  at java.util.Optional.ifPresent(Optional.java:159)
  at org.sonar.server.platform.platformlevel.PlatformLevelStartup$1.doPrivileged(PlatformLevelStartup.java:84)
  at org.sonar.server.user.DoPrivileged.execute(DoPrivileged.java:45)
  at org.sonar.server.platform.platformlevel.PlatformLevelStartup.start(PlatformLevelStartup.java:80)
  at org.sonar.server.platform.Platform.executeStartupTasks(Platform.java:196)
  at org.sonar.server.platform.Platform.access$400(Platform.java:46)
  at org.sonar.server.platform.Platform$1.lambda$doRun$1(Platform.java:121)
  at org.sonar.server.platform.Platform$AutoStarterRunnable.runIfNotAborted(Platform.java:371)
  at org.sonar.server.platform.Platform$1.doRun(Platform.java:121)
  at org.sonar.server.platform.Platform$AutoStarterRunnable.run(Platform.java:355)
  at java.lang.Thread.run(Thread.java:745)
    INFO  web[][o.s.p.StopWatcher] Stopping process
  </code></pre>