title: JanusGraphException: Could not execute operation due to backend exception
body: <p>I'm going crazy because of this exception when starting up JanusGraph.
  It happened between restarts of the gremlin-server, without even touching the configuration files.</p>
  
  <p>This error always appears at the very first startup of gremlin-server.</p>
  
  <p>This is the stack trace from the logs:</p>
  
  <pre><code>6911 [main] INFO  org.janusgraph.diskstorage.log.kcvs.KCVSLog  - Loaded unidentified ReadMarker start time 2019-08-21T17:57:53.794212900Z into org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller@51efb731
  8148 [main] WARN  org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker  - Skipping outdated lock on KeyColumn [k=0x 16-165-160-103-105- 30- 71-114- 97-112-104- 95- 78- 97-109-101- 95- 73-110-100-101-248, c=0x  0] with our rid ( 48- 97- 48- 48- 52- 98- 48- 49- 49- 54- 52- 50- 52- 45- 68- 69- 83- 75- 84- 79- 80- 45- 56- 67- 86- 72- 80- 57- 49- 49) but mismatched timestamp (actual ts 2019-08-21T17:57:54.934645Z, expected ts 2019-08-21T17:57:54.934645300Z)
  8149 [main] ERROR org.janusgraph.graphdb.database.StandardJanusGraph  - Could not commit transaction [1] due to storage exception in system-commit
  org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
      at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56)
      at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91)
      at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:139)
      at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196)
      at org.janusgraph.diskstorage.BackendTransaction.commit(BackendTransaction.java:150)
      at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:716)
      at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1380)
      at org.janusgraph.graphdb.database.management.ManagementSystem.commit(ManagementSystem.java:246)
      at org.janusgraph.graphdb.management.ConfigurationManagementGraph.createIndexIfDoesNotExist(ConfigurationManagementGraph.java:311)
      at org.janusgraph.graphdb.management.ConfigurationManagementGraph.&lt;init&gt;(ConfigurationManagementGraph.java:81)
      at org.janusgraph.graphdb.management.JanusGraphManager.lambda$new$0(JanusGraphManager.java:77)
      at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
      at org.janusgraph.graphdb.management.JanusGraphManager.&lt;init&gt;(JanusGraphManager.java:74)
      at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
      at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.&lt;init&gt;(ServerGremlinExecutor.java:80)
      at org.apache.tinkerpop.gremlin.server.GremlinServer.&lt;init&gt;(GremlinServer.java:122)
      at org.apache.tinkerpop.gremlin.server.GremlinServer.&lt;init&gt;(GremlinServer.java:86)
      at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:345)
  Caused by: org.janusgraph.diskstorage.locking.PermanentLockingException: Permanent locking failure
      at org.janusgraph.diskstorage.locking.AbstractLocker.checkLocks(AbstractLocker.java:359)
      at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.checkAllLocks(ExpectedValueCheckingTransaction.java:175)
      at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.prepareForMutations(ExpectedValueCheckingTransaction.java:154)
      at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:72)
      at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94)
      at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91)
      at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68)
      at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54)
      ... 20 more
  Caused by: org.janusgraph.diskstorage.PermanentBackendException: Read 1 locks with our rid  48- 97- 48- 48- 52- 98- 48- 49- 49- 54- 52- 50- 52- 45- 68- 69- 83- 75- 84- 79- 80- 45- 56- 67- 86- 72- 80- 57- 49- 49 but mismatched timestamps; no lock column contained our timestamp (2019-08-21T17:57:54.934645300Z)
      at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSeniority(ConsistentKeyLocker.java:528)
      at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:454)
      at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:118)
      at org.janusgraph.diskstorage.locking.AbstractLocker.checkLocks(AbstractLocker.java:351)
      ... 27 more
  8152 [main] ERROR org.janusgraph.graphdb.database.StandardJanusGraph  - Could not commit transaction [1] due to exception
  org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
      at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:56)
      at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:91)
      at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:139)
      at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:196)
      at org.janusgraph.diskstorage.BackendTransaction.commit(BackendTransaction.java:150)
      at org.janusgraph.graphdb.database.StandardJanusGraph.commit(StandardJanusGraph.java:716)
      at org.janusgraph.graphdb.transaction.StandardJanusGraphTx.commit(StandardJanusGraphTx.java:1380)
      at org.janusgraph.graphdb.database.management.ManagementSystem.commit(ManagementSystem.java:246)
      at org.janusgraph.graphdb.management.ConfigurationManagementGraph.createIndexIfDoesNotExist(ConfigurationManagementGraph.java:311)
      at org.janusgraph.graphdb.management.ConfigurationManagementGraph.&lt;init&gt;(ConfigurationManagementGraph.java:81)
      at org.janusgraph.graphdb.management.JanusGraphManager.lambda$new$0(JanusGraphManager.java:77)
      at java.base/java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
      at org.janusgraph.graphdb.management.JanusGraphManager.&lt;init&gt;(JanusGraphManager.java:74)
      at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
      at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.&lt;init&gt;(ServerGremlinExecutor.java:80)
      at org.apache.tinkerpop.gremlin.server.GremlinServer.&lt;init&gt;(GremlinServer.java:122)
      at org.apache.tinkerpop.gremlin.server.GremlinServer.&lt;init&gt;(GremlinServer.java:86)
      at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:345)
  Caused by: org.janusgraph.diskstorage.locking.PermanentLockingException: Permanent locking failure
      at org.janusgraph.diskstorage.locking.AbstractLocker.checkLocks(AbstractLocker.java:359)
      at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.checkAllLocks(ExpectedValueCheckingTransaction.java:175)
      at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingTransaction.prepareForMutations(ExpectedValueCheckingTransaction.java:154)
      at org.janusgraph.diskstorage.locking.consistentkey.ExpectedValueCheckingStoreManager.mutateMany(ExpectedValueCheckingStoreManager.java:72)
      at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:94)
      at org.janusgraph.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:91)
      at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:68)
      at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:54)
      ... 20 more
  Caused by: org.janusgraph.diskstorage.PermanentBackendException: Read 1 locks with our rid  48- 97- 48- 48- 52- 98- 48- 49- 49- 54- 52- 50- 52- 45- 68- 69- 83- 75- 84- 79- 80- 45- 56- 67- 86- 72- 80- 57- 49- 49 but mismatched timestamps; no lock column contained our timestamp (2019-08-21T17:57:54.934645300Z)
      at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSeniority(ConsistentKeyLocker.java:528)
      at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:454)
      at org.janusgraph.diskstorage.locking.consistentkey.ConsistentKeyLocker.checkSingleLock(ConsistentKeyLocker.java:118)
      at org.janusgraph.diskstorage.locking.AbstractLocker.checkLocks(AbstractLocker.java:351)
      ... 27 more
  8153 [main] ERROR org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor  - Could not invoke constructor on class org.janusgraph.graphdb.management.JanusGraphManager (defined by the 'graphManager' setting) with one argument of class Settings
  </code></pre>
  
  <p>I'm on windows 10, the version of JanusGraph I'm using is the 0.4.0 and it's configured to use Cassandra 3.11.0 and Elastisearch 6.7.2, both running on docker (while I manually start gremlin-server).</p>
  
  <p>I've already tried cleaning everything from docker (containers, images and volumes).
  I also tried running 'bin/JanusGraph.sh clean-, but with no luck.
  Even downloading again the janusgraph-0.4.0-hadoop2.zip and reconfiguring it from start does nothing.</p>
  
  <p>It goes without saying that if I try to startup the server without touching the original configuration files it works. (But without the ConfigurationManagementGraph)</p>
  
  <p>I don't understand how it could break without touching anything.</p>
  
  <p>This is the gremlin-server configuration:</p>
  
  <pre><code>host: 0.0.0.0
  port: 8182
  scriptEvaluationTimeout: 180000
  # channelizer: org.apache.tinkerpop.gremlin.server.channel.WebSocketChannelizer
  channelizer: org.janusgraph.channelizers.JanusGraphWebSocketChannelizer
  graphManager: org.janusgraph.graphdb.management.JanusGraphManager
  graphs: {
    #graph: conf/gremlin-server/janusgraph-cql-es-server.properties,
    ConfigurationManagementGraph: conf/gremlin-server/janusgraph-cql-es-server-configured.properties,
  }
  scriptEngines: {
    gremlin-groovy: {
      plugins: { org.janusgraph.graphdb.tinkerpop.plugin.JanusGraphGremlinPlugin: {},
                 org.apache.tinkerpop.gremlin.server.jsr223.GremlinServerGremlinPlugin: {},
                 org.apache.tinkerpop.gremlin.tinkergraph.jsr223.TinkerGraphGremlinPlugin: {},
                 org.apache.tinkerpop.gremlin.jsr223.ImportGremlinPlugin: {classImports: [java.lang.Math], methodImports: [java.lang.Math#*]},
                 org.apache.tinkerpop.gremlin.jsr223.ScriptFileGremlinPlugin: {files: [scripts/init.groovy]}}}}
  serializers:
    - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
    - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV3d0, config: { serializeResultToString: true }}
    - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV3d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
  #  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphBinaryMessageSerializerV1 }
  #  - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphBinaryMessageSerializerV1, config: { serializeResultToString: true }} 
    # Older serialization versions for backwards compatibility:
    - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
    - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoLiteMessageSerializerV1d0, config: {ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
    - { className: org.apache.tinkerpop.gremlin.driver.ser.GryoMessageSerializerV1d0, config: { serializeResultToString: true }}
    - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV2d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistry] }}
    - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerGremlinV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistryV1d0] }}
    - { className: org.apache.tinkerpop.gremlin.driver.ser.GraphSONMessageSerializerV1d0, config: { ioRegistries: [org.janusgraph.graphdb.tinkerpop.JanusGraphIoRegistryV1d0] }}
  processors:
    - { className: org.apache.tinkerpop.gremlin.server.op.session.SessionOpProcessor, config: { sessionTimeout: 28800000 }}
    - { className: org.apache.tinkerpop.gremlin.server.op.traversal.TraversalOpProcessor, config: { cacheExpirationTime: 600000, cacheMaxSize: 1000 }}
  metrics: {
    consoleReporter: {enabled: true, interval: 180000},
    csvReporter: {enabled: true, interval: 180000, fileName: /tmp/gremlin-server-metrics.csv},
    jmxReporter: {enabled: true},
    slf4jReporter: {enabled: true, interval: 180000},
    gangliaReporter: {enabled: false, interval: 180000, addressingMode: MULTICAST},
    graphiteReporter: {enabled: false, interval: 180000}}
  maxInitialLineLength: 4096
  maxHeaderSize: 8192
  maxChunkSize: 8192
  maxContentLength: 65536
  maxAccumulationBufferComponents: 1024
  resultIterationBatchSize: 64
  writeBufferLowWaterMark: 32768
  writeBufferHighWaterMark: 65536
  </code></pre>
  
  <p>This is the janus graph configuration (I removed the comments for brevity):</p>
  
  <pre><code>gremlin.graph=org.janusgraph.core.ConfiguredGraphFactory
  
  graph.graphname=ConfigurationManagementGraph
  
  storage.backend=cql
  
  storage.hostname=127.0.0.1
  
  storage.cql.keyspace=janusgraph
  
  cache.db-cache = true
  
  cache.db-cache-time = 180000
  
  cache.db-cache-size = 0.25
  
  index.search.backend=elasticsearch
  
  index.search.hostname=127.0.0.1
  
  index.search.elasticsearch.client-only=true
  </code></pre>
  
  <p>This is the script gremlin server runs at startup:</p>
  
  <pre><code>def globals = [:]
  
  
  def getGraph() {
      def graphNames =  ConfiguredGraphFactory.getGraphNames();
      def graphMaps = [:];
      for (graphName in graphNames) {
          def g = ConfiguredGraphFactory.open(graphName);
          graphMaps.put(graphName, g.traversal())
      }
      return graphMaps;
  }
  
  globals &lt;&lt; getGraph()
  </code></pre>
  
  <p>In the docker-compose console I see no exceptions, the last thing before gremlin-server shows the error is the following line:</p>
  
  <pre><code>cassandra        | INFO  [MigrationStage:1] 2019-08-21 17:57:53,725 ColumnFamilyStore.java:430 - Initializing janusgraph.system_properties_lock_
  </code></pre>
  
  <p>Please has anyone any idea what might be causing this problem?</p>
  
  <p>Thank you!</p>