title: Cassandra Cluster in AWS Multi Region VPC
body: <p>I am trying to achieve the following schema for my Cassandra Cluster :</p>
  
  <ul>
  <li>1 VPC in 4 different AWS Regions linked together with IPSec instances.</li>
  <li>1 Cassandra Cluster made of 4 nodes, 1 in each VPC</li>
  <li>Nodes communicating together with private IP in VPC (10.0.0.0/8)</li>
  <li>Cassandra data accessible through REST API of my own from Public IP.</li>
  </ul>
  
  <p>So far, I've been able to achieve the configuration of the cluster, installing OpsCenter and checking that every agent is working fine. (For reference, I used GossipPropertyFileSnitch and have put "dc=us-west, rack=1b" in the rack config.</p>
  
  <p>My problem is that my HTTP API is slow and is getting Timeout way too much. I've been trying to run some import scripts (over HTTP that inserts in Cassandra via CQL Driver) and keep getting this type of error :</p>
  
  <blockquote>
    <p>Error while executing batch:com.google.common.util.concurrent.UncheckedExecutionException: java.lang.Runtim
    eException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.</p>
  </blockquote>
  
  <p>For reference, the corresponding error in system.log is : </p>
  
  <pre><code>ERROR [SharedPool-Worker-1] 2015-03-04 19:25:39,598 ErrorMessage.java:243 - Unexpected exception during request
  com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
  at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na]
  at com.google.common.cache.LocalCache.get(LocalCache.java:3934) ~[guava-16.0.jar:na]
  at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na]
  at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na]
  at org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:56) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.auth.Auth.getPermissions(Auth.java:78) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.service.ClientState.authorize(ClientState.java:352) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:250) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:244) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:228) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.cql3.statements.ModificationStatement.checkAccess(ModificationStatement.java:128) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.cql3.statements.BatchStatement.checkAccess(BatchStatement.java:86) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.cql3.QueryProcessor.processBatch(QueryProcessor.java:500) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.transport.messages.BatchMessage.execute(BatchMessage.java:215) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:439) [apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:335) [apache-cassandra-2.1.3.jar:2.1.3]
  at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-all-4.0.23.Final.jar:4.0.23.Final]
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) [netty-all-4.0.23.Final.jar:4.0.23.Final]
  at io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) [netty-all-4.0.23.Final.jar:4.0.23.Final]
  at io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) [netty-all-4.0.23.Final.jar:4.0.23.Final]
  at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_31]
  at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) [apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.3.jar:2.1.3]
  at java.lang.Thread.run(Unknown Source) [na:1.8.0_31]
  Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
  at org.apache.cassandra.auth.Auth.selectUser(Auth.java:279) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.auth.Auth.isSuperuser(Auth.java:100) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:50) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:67) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:82) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.auth.PermissionsCache$1.load(PermissionsCache.java:79) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524) ~[guava-16.0.jar:na]
  at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317) ~[guava-16.0.jar:na]
  at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280) ~[guava-16.0.jar:na]
  at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195) ~[guava-16.0.jar:na]
  ... 23 common frames omitted
  Caused by: org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - received only 0 responses.
  at org.apache.cassandra.service.ReadCallback.get(ReadCallback.java:103) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.service.AbstractReadExecutor.get(AbstractReadExecutor.java:139) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:1338) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.service.StorageProxy.readRegular(StorageProxy.java:1265) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:1188) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:253) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:206) ~[apache-cassandra-2.1.3.jar:2.1.3]
  at org.apache.cassandra.auth.Auth.selectUser(Auth.java:268) ~[apache-cassandra-2.1.3.jar:2.1.3]
  ... 32 common frames omitted
  </code></pre>
  
  <p>It did work sometime, and I was even able to connect to DevCenter and actually see my data. But it's failing too much.</p>
  
  <p>My temporary solution was to enable communication on the public IP of each instance and still have them work on private IP together. I'm doing my import at the moment.</p>
  
  <p>Now I'm still wondering :</p>
  
  <ul>
  <li><strong>Is my solution viable (Cassandra through VPC + IPSec)</strong></li>
  <li><strong>If not, is the SSL node-to-node viable ?</strong></li>
  <li><strong>Where is this timeout coming from ?</strong></li>
  </ul>
  
  <p>Thank you for your help.</p>