Hot questions for Using Cassandra in cluster computing

Top Java Programmings / Cassandra / cluster computing

Question:

I am writing one java application that uses cassandra cql driver to get/set data to cassandra. I have 2 nodes in my cluster and want to add both of them in addContactPoints so that requests are handled in case of failover or load sharing.

Can someone tell me more about addContactPoints(String... addresses) as in how does it works? Will only 1 node be used till its down or both of them will be used to balance the nodes? I am not clear with the working on this function and could not find much clarity over web. Also in case if one node is not reachable will it automatically use the other ip or some other property also needs to be set?

Thanks!


Answer:

From the javadoc

Note that by default (that is, unless you use the withLoadBalancingPolicy) method of this builder), the first succesfully contacted host will be use to define the local data-center for the client. If follows that if you are running Cassandra in a multiple data-center setting, it is a good idea to only provided contact points that are in the same datacenter than the client, or to provide manually the load balancing policy that suits your need.

The default load balancing policy is DCAwareRoundRobinPolicy.

In other words, this policy guarantees that no host in a remote data center will be queried unless no host in the local data center can be reached.

It will keep trying other hosts if others can't be reached.

Question:

I'm trying to setup a 2-node cassandra-2.1 cluster with the following node configurations:

Cluster Name: 'Cluster1'
num_tokens: 256
listen_address: 10.20.0.52/10.20.0.53
rpc_address: 10.20.0.52/10.20.0.53
class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
    # seeds is actually a comma-delimited list of addresses.
    # Ex: "<ip1>,<ip2>,<ip3>"
    - seeds: "10.20.0.52"

I first start the seed node (52) then, I check nodetool status and returns data only for 52. But then I boot (53) and nodetool status throws the following exception after a few seconds:

-- StackTrace --
java.lang.RuntimeException: No nodes present in the cluster. Has this node finished starting up?
        at org.apache.cassandra.dht.Murmur3Partitioner.describeOwnership(Murmur3Partitioner.java:131)
        at org.apache.cassandra.service.StorageService.getOwnership(StorageService.java:3912)
        at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
        at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
        at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
        at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
        at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83)
        at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1443)
        at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1307)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1399)
        at javax.management.remote.rmi.RMIConnectionImpl.getAttribute(RMIConnectionImpl.java:637)
        at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:323)
        at sun.rmi.transport.Transport$1.run(Transport.java:200)
        at sun.rmi.transport.Transport$1.run(Transport.java:197)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$250(TCPTransport.java:683)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$1/1165999373.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)

but on the non-seed node (53) it returns the standard output with details only for itself (53). nodetool gossipinfo on the seed node (52) returns information about both nodes:

/10.20.0.52
  generation:1439824481
  heartbeat:2433
  SCHEMA:500091e4-e8ab-303d-9111-8cca7edff2d0
  HOST_ID:2d78ed48-13e8-4fc5-ac55-8b2a6d00c8c5
  NET_VERSION:8
  RELEASE_VERSION:2.1.8-SNAPSHOT
  STATUS:NORMAL,-1091407767707699731
  RPC_ADDRESS:10.20.0.52
  SEVERITY:0.5025125741958618
  DC:DC1
  LOAD:2524926.0
  RACK:RAC1
  INTERNAL_IP:10.20.0.52
/10.20.0.53
  generation:1439824502
  heartbeat:2376
  SCHEMA:500091e4-e8ab-303d-9111-8cca7edff2d0
  NET_VERSION:8
  HOST_ID:2d78ed48-13e8-4fc5-ac55-8b2a6d00c8c5
  RELEASE_VERSION:2.1.8-SNAPSHOT
  STATUS:NORMAL,-1091407767707699731
  RPC_ADDRESS:10.20.0.53
  SEVERITY:0.0
  DC:DC1
  LOAD:2603302.0
  RACK:RAC1
  INTERNAL_IP:10.20.0.53

but on the non-seed node it only displays information about itself and does not include the seed node (52).

Another discrepancy between the state/information about the 2 nodes is the output of nodetool netstats which for the seed node (52) shows:

ubuntu@52:~$ nodetool netstats 
Mode: NORMAL
Not sending any streams.
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name                    Active   Pending      Completed
Commands                        n/a         0              0
Responses                       n/a         0           1135

while for the non-seed node (53) the number of requests completed is double of that of the seed node:

ubuntu@53:~$ nodetool netstats 
Mode: NORMAL
Not sending any streams.
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name                    Active   Pending      Completed
Commands                        n/a         0              0
Responses                       n/a         0           2388

Source code Given the stacktrace, I tried to insert some flags and print what appears to be causing the error on L206 Murmur3Partitioner.java when describeOwnership method is called: - the method is called when the seed node is booted - the method is called when the non-seed node is bootstrapped

both times the list of tokes (or sortedTokens) is exactly the same, yet the iterator is empty and triggers the error in the title.

Note: the relvant ports (7000,7001) on both nodes (52,53) are open.

Update #1: so, I found out (thanks to irc #cassandra channel) that if two nodes have the same tokes a conflict is created and one will fail to bootstrap.

To address this I tried the following: cqlsh> DROP KEYSPACE ycsb ;

which didn't fix the issue - nodetool ring still showed the same tokens corresponding to the non-seed node; I also flushed the changes after closing cqlsh. Then:

sudo rm -rf /var/lib/cassandra/data/*
sudo rm -rf /var/lib/cassandra/commitlog/*
sudo rm -rf /var/lib/cassandra/saved_caches/*

which still didn't reduce or change the tokes that show up in nodetool ring.

Any guidance is appreciated.


Answer:

The culprit appears to have been the ports and firewall rules which wouldn't allow for nodes to establish bidirectional symmetric connections in order to exchange tokens residing on each node. The troubleshooting steps taken were: 1) nestat -l on both nodes to see which ports are open/listening; 2) nmap from one node to another to scan open ports. 3) nodetool ring to compare the tokens on both nodes 4) TRACE logging level set in logback.xml and output either in a separate log file or to stderr

I also recommend discussing your issues with #cassandra IRC-channel. The folks there are very knowledgeable and can help in almost real-time.

Hope it helps!

Question:

The token value that I'm using:

initial_token: 85070591730234615865843651857942052864

Is causing the following Java exception when I try to start Cassandra:

Exception (java.lang.IllegalArgumentException) encountered during startup: Invalid token for Murmur3Partitioner. Got 85070591730234615865843651857942052864 but expected a long value (unsigned 8 bytes integer).
java.lang.IllegalArgumentException: Invalid token for Murmur3Partitioner. Got 85070591730234615865843651857942052864 but expected a long value (unsigned 8 bytes integer).
    at org.apache.cassandra.dht.Murmur3Partitioner$2.fromString(Murmur3Partitioner.java:333)
    at org.apache.cassandra.dht.Murmur3Partitioner$2.validate(Murmur3Partitioner.java:317)
    at org.apache.cassandra.config.DatabaseDescriptor.applyInitialTokens(DatabaseDescriptor.java:885)
    at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:321)
    at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:141)
    at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:646)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:581)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:735)
ERROR [main] 2017-03-01 19:57:15,861 CassandraDaemon.java:752 - Exception encountered during startup
java.lang.IllegalArgumentException: Invalid token for Murmur3Partitioner. Got 85070591730234615865843651857942052864 but expected a long value (unsigned 8 bytes integer).
    at org.apache.cassandra.dht.Murmur3Partitioner$2.fromString(Murmur3Partitioner.java:333) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.dht.Murmur3Partitioner$2.validate(Murmur3Partitioner.java:317) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.config.DatabaseDescriptor.applyInitialTokens(DatabaseDescriptor.java:885) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:321) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:141) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:646) [apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:581) [apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:735) [apache-cassandra-3.10.jar:3.10]

Cassandra starts fine on my first node with initial_token: 0

What am I doing wrong?

Any help is much appreciated!


Answer:

My main question would be why are you trying to set it?

That token (85070591730234615865843651857942052864) is likely for the random partitioner, not murmur partitioner. Can check other nodes in your cluster but perhaps set in the cassandra.yaml:

partitioner: org.apache.cassandra.dht.RandomPartitioner

Murmur3 partitioner is better idea though, not sure where you got that token or why you expect it to work. Is better to let it assign tokens itself if your just adding nodes. Just do not set it.

# initial_token: