title: Having trouble getting DataStax OpsCenter 4.0.3 to connect to Cassandra 2.0.4 cluster
body: <p>I have a 1-node C* 2.0.4 cluster running and nodetool status shows a healthy cluster.</p>
  
  <p>I then installed OpsCenter 4.0.3 on a separate machine on the same network using 'sudo yum install opscenter-free'.</p>
  
  <p>In the opscenterd.conf file, I set the interface = 'public IP of OpsCenter server' and started the OpsCenter server.</p>
  
  <p>I was then able to see the OpsCenter webpage and clicked on Use Existing Cluster. </p>
  
  <p>Under the Add Cluster interface, I typed in the rpc_address for the 1-node Cassandra cluster. OpsCenter accepted it and showed the cluster name correctly on the next page.</p>
  
  <p>However, none of the graphs in OpsCenter load and I see the error: 0 of 0 agents connected. I also see a blinking red X with against a plug icon on the top.</p>
  
  <p>Firewall is currently turned off in CentOS on both the OpsCenter and C* nodes.</p>
  
  <p>How do I get OpsCenter to properly connect to the C* node?</p>
  
  <p>Here's what the OpsCenter log shows (note: I replaced the IP with A.B.C.D):</p>
  
  <pre><code>2014-01-30 06:43:37+0000 [Dog]  WARN: Unable to collect datacenter, rack information: Failed query to http://A.B.C.D:61621/cluster/topology?node_ip=A.B.C.D : Connection was refused by other side: 111: Connection refused.
  2014-01-30 06:45:37+0000 [Dog]  WARN: HTTP request http://A.B.C.D:61621/cluster/topology?node_ip=A.B.C.D failed: Connection was refused by other side: 111: Connection refused.
  2014-01-30 06:45:37+0000 [Dog]  WARN: Unable to collect datacenter, rack information: Failed query to http://A.B.C.D:61621/cluster/topology?node_ip=A.B.C.D : Connection was refused by other side: 111: Connection refused.
  </code></pre>
  
  <p>On the Cassandra node, everything looks healthy:</p>
  
  <pre><code>[root@cassandra01 ~]# nodetool status
  Datacenter: datacenter1
  =======================
  Status=Up/Down
  |/ State=Normal/Leaving/Joining/Moving
  --  Address        Load       Tokens  Owns   Host ID                               Rack
  UN  A.B.C.D  158.27 KB  256     100.0%  bc560cd6-a20d-4b36-99ca-ed477dc939b5  rack1
  </code></pre>
  
  <p>However, I can't curl that URL that OpsCenter is trying to get to:</p>
  
  <pre><code>[root@cassandra01 ~]# curl http://A.B.C.D:61621/cluster/topology?node_ip=A.B.C.D
  curl: (7) couldn't connect to host
  </code></pre>
  
  <p>SSL is turned off (default settings) in the OpsCenterd.conf file.</p>
  
  <p>Here is what I see at the following URL http://<em>Public IP of OpsCenter</em>:8888/Dog/nodes</p>
  
  <pre><code>[{"load": null, "has_jna": false, "vnodes": true, "devices": {"saved_caches": null, "commitlog": null, "other": null, "data": null}, "task_progress": {}, "node_ip": "A.B.C.D", "network_interfaces": null, "ec2": {}, "node_version": {}, "dc": null, "node_name": null, "num_procs": null, "streaming": {}, "token": "5743408169174478324", "data_held": null, "mode": "unknown", "rpc_ip": "10.183.132.141", "partitions": {"saved_caches": null, "commitlog": null, "other": null, "data": null}, "os": null, "rack": null, "last_seen": 0}]
  </code></pre>
  
  <p>Any ideas of how to fix this?</p>
  
  <p>Note, in Cassandra's YAML file, the rpc_server_type is set to sync.</p>
  
  <hr>
  
  <p>Update:</p>
  
  <p>I also tried to manually install the OpsCenter agent on the C* node with 'yum install datastax-agent' and then edited the address.yaml file with the following settings:</p>
  
  <pre><code>stomp_interface: 'public ip of machine opscenterd is running on (public IP)'
  local_interface: 'listen_address in cassandra.yaml (public IP)'
  agent_rpc_interface: 'rpc address in cassandra.yaml (private IP network)'
  agent_rpc_broadcast_address: 'private network IP, same network at rpc address'
  </code></pre>
  
  <p>I tried a few different settings for the address.yaml file and none of them worked. For example, I tried to only set the stop_interface and deleted the other 3 lines. Didn't work. I also tried to set just to stop and local interfaces and that also didn't work.</p>
  
  <p>When I now start the datastax agent with 'service datastax-agent start', suddenly the Cassandra service crashes:</p>
  
  <p>[root@cassandra01 ~]# sudo service cassandra status
  cassandra dead but pid file exists</p>
  
  <p>When the C* service crashes, the opscenter agent stays up and running. If I stop the agent service and start the C* service again (sudo service cassandra status), then the C* starts back up successfully and nodetool status shows a healthy 1-node cluster. But the as soon as I start the agent service, the C* service suddenly crashes again. All the different settings I tried in the address.yaml file causes this same behavior.</p>
  
  <p>Ideally, I'd rather not install the Agent manually and would like to just push it's installer from the OpsCenter GUI onto the C* node, but since that wasn't worked I tried to manually install the agent and connect it to OpsCenter, but this isn't working either unfortunately.</p>
  
  <p>And sometimes I see this as well on the Cassandra node when the Cassandra service crashes:
  [root@cassandra01 ~]# sudo service cassandra stoplog4j:WARN No appenders could be found for logger (org.eclipse.jetty.util.log).
  log4j:WARN Please initialize the log4j system properly.
  log4j:WARN See <a href="http://logging.apache.org/log4j/1.2/faq.html#noconfig" rel="nofollow">http://logging.apache.org/log4j/1.2/faq.html#noconfig</a> for more info.
  Usage: cassandra start|stop|status|restart|reload</p>
  
  <p>Here is what the Cassandra node's log4j-server.properties has in it:</p>
  
  <pre><code>log4j.rootLogger=INFO,stdout,R
  
  # stdout
  log4j.appender.stdout=org.apache.log4j.ConsoleAppender
  log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
  log4j.appender.stdout.layout.ConversionPattern=%5p %d{HH:mm:ss,SSS} %m%n
  
  # rolling log file
  log4j.appender.R=org.apache.log4j.RollingFileAppender
  log4j.appender.R.maxFileSize=20MB
  log4j.appender.R.maxBackupIndex=50
  log4j.appender.R.layout=org.apache.log4j.PatternLayout
  log4j.appender.R.layout.ConversionPattern=%5p [%t] %d{ISO8601} %F (line %L) %m%n
  # Edit the next line to point to your logs directory
  log4j.appender.R.File=/var/log/cassandra/system.log
  
  # Application logging options
  #log4j.logger.org.apache.cassandra=DEBUG
  #log4j.logger.org.apache.cassandra.db=DEBUG
  #log4j.logger.org.apache.cassandra.service.StorageProxy=DEBUG
  
  # Adding this to avoid thrift logging disconnect errors.
  log4j.logger.org.apache.thrift.server.TNonblockingServer=ERROR
  </code></pre>
  
  <p>And finally, here is what the agent.log from the OpsCenter agent running on the Cassandra node shows:</p>
  
  <pre><code>nohup: ignoring input
  Starting DataStax agent monitor datastax_agent_monitor
   INFO [main] 2014-01-30 08:24:59,104 Loading conf files: /var/lib/datastax-agent /conf/address.yaml
   INFO [main] 2014-01-30 08:24:59,261 Java vendor/version: Java HotSpot(TM) 64-Bi t Server VM/1.7.0_25
   INFO [main] 2014-01-30 08:24:59,546 Default config values: {:rollups300_ttl 241 9200, :settings_cf "settings", :agent_rpc_interface "10.183.132.141", :my_channe l_prefix "/agent", :poll_period 60, :kerberos_hostname nil, :storage_dc nil, :th rift_conn_timeout 10000, :thrift_max_frame_size 15728640, :rollups60_ttl 604800,  :stomp_port 61620, :shorttime_interval 10, :longtime_interval 300, :private-con f-props ["initial_token" "listen_address" "broadcast_address" "rpc_address"], :t hrift_port 9160, :async_retry_timeout 5, :agent-conf-group "global-cluster-agent -group", :jmx_host "127.0.0.1", :ec2_metadata_api_host "169.254.169.254", :metri cs_enabled 1, :async_queue_size 5000, :autodiscovery_interval 120, :rollups7200_ ttl 31536000, :autodiscovery_enabled true, :thrift_ssl_truststore nil, :rollup_s napshot_period 300, :is_package true, :monitor_command "/usr/share/datastax-agen t/bin/datastax_agent_monitor", :thrift_socket_timeout 5000, :cassandra_log_locat ion "/var/log/cassandra/system.log", :local_interface "23.253.64.169", :jmx_port  7199, :jmx_metrics_threadpool_size 4, :use_ssl 0, :rollups86400_ttl -1, :nodede tails_threadpool_size 3, :api_port 61621, :kerberos_service nil, :kerberos_clien t_principal nil, :jmx_thread_pool_size 5, :production 1, :stomp_interface "166.7 8.186.184", :storage_keyspace "OpsCenter", :rollup_snapshot_threshold 300, :thri ft_ssl_truststore_type "JKS", :realtime_interval 5}
   INFO [main] 2014-01-30 08:24:59,554 Waiting for the config from OpsCenter
   INFO [main] 2014-01-30 08:24:59,559 Using 23.253.64.169 as the cassandra broadc ast address
   INFO [main] 2014-01-30 08:24:59,568 New JMX connection (127.0.0.1:7199)
  ERROR [main] 2014-01-30 08:25:00,019 Error connecting via JMX: java.io.IOExcepti on: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException  [Root exception is java.rmi.ConnectException: Connection refused to host: 127.0. 0.1; nested exception is:
          java.net.ConnectException: Connection refused]
   INFO [main] 2014-01-30 08:25:00,414 cassandra RPC address is  nil
   INFO [main] 2014-01-30 08:25:00,418 agent RPC broadcast address is  10.183.132. 141
   INFO [main] 2014-01-30 08:25:00,474 Clearing ssl.truststore
   INFO [main] 2014-01-30 08:25:00,475 Clearing ssl.truststore.password
   INFO [main] 2014-01-30 08:25:00,476 Setting ssl.store.type to JKS
   INFO [main] 2014-01-30 08:25:00,477 Clearing kerberos.service.principal.name
   INFO [main] 2014-01-30 08:25:00,480 Clearing kerberos.principal
   INFO [main] 2014-01-30 08:25:00,480 Clearing kerberos.useTicketCache
   INFO [main] 2014-01-30 08:25:00,481 Clearing kerberos.ticketCache
   INFO [main] 2014-01-30 08:25:00,487 Clearing kerberos.useKeyTab
   INFO [main] 2014-01-30 08:25:00,487 Clearing kerberos.keyTab
   INFO [main] 2014-01-30 08:25:00,487 Clearing kerberos.renewTGT
   INFO [main] 2014-01-30 08:25:00,488 Clearing kerberos.debug
   INFO [main] 2014-01-30 08:25:00,495 Starting Stomp
   INFO [main] 2014-01-30 08:25:00,495 SSL communication is disabled
   INFO [main] 2014-01-30 08:25:00,495 Creating stomp connection to 166.78.186.184 :61620
   INFO [thrift-init] 2014-01-30 08:25:00,521 Connecting to Cassandra cluster: 23. 253.64.169 (port 9160)
   INFO [StompConnection receiver] 2014-01-30 08:25:00,536 Reconnecting in 0s.
   INFO [StompConnection receiver] 2014-01-30 08:25:00,561 Connected to 166.78.186 .184:61620
   INFO [thrift-init] 2014-01-30 08:25:00,619 Downed Host Retry service started wi th queue size -1 and retry delay 10s
   INFO [thrift-init] 2014-01-30 08:25:00,662 Registering JMX me.prettyprint.cassa ndra.service_Agent Cluster:ServiceType=hector,MonitorType=hector
   INFO [main] 2014-01-30 08:25:00,732 Starting Jetty server: {:port 61621, :host  "10.183.132.141", :ssl? false, :join? false}
  ERROR [thrift-init] 2014-01-30 08:25:00,885 MARK HOST AS DOWN TRIGGERED for host  23.253.64.169(23.253.64.169):9160
  ERROR [thrift-init] 2014-01-30 08:25:00,886 Pool state on shutdown: &lt;ConcurrentC assandraClientPoolByHost&gt;:{23.253.64.169(23.253.64.169):9160}; IsActive?: true;  Active: 0; Blocked: 1; Idle: 0; NumBeforeExhausted: 1
   INFO [thrift-init] 2014-01-30 08:25:00,887 Shutdown triggered on &lt;ConcurrentCas sandraClientPoolByHost&gt;:{23.253.64.169(23.253.64.169):9160}
   INFO [thrift-init] 2014-01-30 08:25:00,901 Shutdown complete on &lt;ConcurrentCass andraClientPoolByHost&gt;:{23.253.64.169(23.253.64.169):9160}
   INFO [thrift-init] 2014-01-30 08:25:00,902 Host detected as down was added to r etry queue: 23.253.64.169(23.253.64.169):9160
   WARN [thrift-init] 2014-01-30 08:25:00,914 Could not fullfill request on this h ost null
   WARN [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] 2 014-01-30 08:25:00,910 Downed 23.253.64.169(23.253.64.169):9160 host still appea rs to be down: Unable to open transport to 23.253.64.169(23.253.64.169):9160 , j ava.net.ConnectException: Connection refused
   WARN [thrift-init] 2014-01-30 08:25:00,926 Exception:
  me.prettyprint.hector.api.exceptions.HectorTransportException: Unable to open tr ansport to 23.253.64.169(23.253.64.169):9160 , java.net.ConnectException: Connec tion refused
          at me.prettyprint.cassandra.connection.client.HThriftClient.open(HThrift Client.java:180)
          at me.prettyprint.cassandra.connection.client.HThriftClient.open(HThrift Client.java:38)
          at me.prettyprint.cassandra.connection.ConcurrentHClientPool.createClien t(ConcurrentHClientPool.java:162)
          at me.prettyprint.cassandra.connection.ConcurrentHClientPool.borrowClien t(ConcurrentHClientPool.java:94)
          at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFai lover(HConnectionManager.java:250)
          at me.prettyprint.cassandra.service.AbstractCluster.describeClusterName( AbstractCluster.java:155)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
          at java.lang.reflect.Method.invoke(Unknown Source)
          at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
          at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
          at clj_hector.core$cluster_name.invoke(core.clj:40)
          at opsagent.cassandra$setup_cassandra$f__353__auto____900$fn__920.invoke (cassandra.clj:360)
          at opsagent.cassandra$setup_cassandra$f__353__auto____900.invoke(cassand ra.clj:358)
          at clojure.lang.AFn.run(AFn.java:24)
          at java.lang.Thread.run(Unknown Source)
  Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectExce ption: Connection refused
          at org.apache.thrift.transport.TSocket.open(TSocket.java:183)
          at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.ja va:81)
          at me.prettyprint.cassandra.connection.client.HThriftClient.open(HThrift Client.java:174)
          ... 16 more
  Caused by: java.net.ConnectException: Connection refused
          at java.net.PlainSocketImpl.socketConnect(Native Method)
          at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
          at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
          at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
          at java.net.SocksSocketImpl.connect(Unknown Source)
          at java.net.Socket.connect(Unknown Source)
          at org.apache.thrift.transport.TSocket.open(TSocket.java:178)
          ... 18 more
  ERROR [thrift-init] 2014-01-30 08:25:00,965 Error when performing thrift operati on:
  me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down . Retry burden pushed out to client.
          at me.prettyprint.cassandra.connection.HConnectionManager.getClientFromL BPolicy(HConnectionManager.java:395)
          at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFai lover(HConnectionManager.java:249)
          at me.prettyprint.cassandra.service.AbstractCluster.describeClusterName( AbstractCluster.java:155)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
          at java.lang.reflect.Method.invoke(Unknown Source)
          at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
          at clojure.lang.Reflector.invokeNoArgInstanceMember(Reflector.java:298)
          at clj_hector.core$cluster_name.invoke(core.clj:40)
          at opsagent.cassandra$setup_cassandra$f__353__auto____900$fn__920.invoke (cassandra.clj:360)
          at opsagent.cassandra$setup_cassandra$f__353__auto____900.invoke(cassand ra.clj:358)
          at clojure.lang.AFn.run(AFn.java:24)
          at java.lang.Thread.run(Unknown Source)
   INFO [StompConnection receiver] 2014-01-30 08:25:01,024 Got new config from Ops Center: {:kerberos_use_keytab true, :rollups300_ttl 2419200, :kerberos_use_ticke t_cache true, :rollups60_ttl 604800, :thrift_port 9160, :ec2_metadata_api_host " 169.254.169.254", :metrics_enabled 1, :rollups7200_ttl 31536000, :thrift_ssl_tru ststore nil, :metrics_ignored_column_families "", :cassandra_log_location "/var/ log/cassandra/system.log", :thrift_rpc_interface "10.183.132.141", :thrift_ssl_t ruststore_password nil, :jmx_port 7199, :provisioning 0, :use_ssl 0, :kerberos_d ebug false, :rollups86400_ttl -1, :api_port "61621", :storage_keyspace "OpsCente r", :kerberos_renew_tgt true, :metrics_ignored_solr_cores "", :thrift_ssl_trusts tore_type "JKS", :metrics_ignored_keyspaces "system, system_traces, system_auth,  dse_auth, OpsCenter", :rollup_subscriptions [], :cassandra_install_location ""}
   INFO [StompConnection receiver] 2014-01-30 08:25:01,030 Starting up agent colle ction.
   INFO [StompConnection receiver] 2014-01-30 08:25:01,040 New JMX connection (127 .0.0.1:7199)
  ERROR [StompConnection receiver] 2014-01-30 08:25:01,073 Error connecting via JM X: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceU navailableException [Root exception is java.rmi.ConnectException: Connection ref used to host: 127.0.0.1; nested exception is:
          java.net.ConnectException: Connection refused]
   INFO [Jetty] 2014-01-30 08:25:01,160 Jetty server started
   INFO [StompConnection receiver] 2014-01-30 08:25:01,188 Starting OS metric coll ectors (Linux)
   INFO [StompConnection receiver] 2014-01-30 08:25:01,199 Starting Cassandra JMX  metric collectors
   INFO [install-location-finder] 2014-01-30 08:25:01,250 New JMX connection (127. 0.0.1:7199)
   INFO [StompConnection receiver] 2014-01-30 08:25:01,252 New JMX connection (127 .0.0.1:7199)
  ERROR [install-location-finder] 2014-01-30 08:25:01,261 Error connecting via JMX : java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUn availableException [Root exception is java.rmi.ConnectException: Connection refu sed to host: 127.0.0.1; nested exception is:
          java.net.ConnectException: Connection refused]
  </code></pre>