Hot questions for Using Cassandra in jmx

Question:

I know that nodetool describecluster can give me information about schema disagreements in cassandra . But, i am looking to see if that information is available via JMX .If it is available can some one point me fully-qualified package/metric name for this ?

I am trying setup alert for this using Prometheus .


Answer:

Yes, you can monitor it with JMX:

ObjectName: org.apache.cassandra.db:type=StorageProxy
Attribute:  SchemaVersions

This will return a key/value pair, with a format similar to:

eaceacea-eac7-eac6-eaca-eac75275b=[172.1.xx.xx, 172.1.xx.xx, 172.1.xx.xx]

All the nodes in the same cluster should match the UUID returned.

Question:

Our Cassandra 2.2 cluster (on CentOS 7) is working fine except one thing. As soon as I put LOCAL_JMX=no in cassandra-env.sh, Cassandra fails to start with the following error:

Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 7199; nested exception is:
    java.net.BindException: Address already in use

The relevant configuration is as follows:

  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
  JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStore=/etc/pki/cassandra/keys/.keystore"
  JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStorePassword=password1"
  JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStore=/etc/pki/cassandra/certs/.truststore"
  JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStorePassword=password2"
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.need.client.auth=true"
  JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.registry.ssl=true"

I indeed verified if there isn't anything running on the port and it isn't. If I change the port, the same result. What helped was to use different ports for jmxremote.port and for jmxremote.rmi.port, although it is against recommendations in manuals. Unfortunately even that I wasn't able to use nodetool getting NoSuchObjectException: 'no such object in table' exception.

Any ideas? Many thanks.


Answer:

I have found the solution. The problem was in the following line:

JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"

When I switched it to true, it was possible to start Cassandra with the settings mentioned above (using a common port).

However it didn't mean the problems were over, because while using nodetool I started to get: nodetool: Failed to connect to - ConnectIOException: 'non-JRMP server at remote endpoint'

The final solution is that the problem is in SSL. The protocol does not support it for JMX. Hence the only solution is to switch it off and the working configuration is as follows:

JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.port=$JMX_PORT"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl=false"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password"
JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStore=/etc/pki/cassandra/keys/.keystore"
JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.keyStorePassword=password1"
JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStore=/etc/pki/cassandra/certs/.truststore"
JVM_OPTS="$JVM_OPTS -Djavax.net.ssl.trustStorePassword=password2"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.ssl.need.client.auth=false"
JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.registry.ssl=false"

Then everything works like a charm.

Question:

I'm trying to access the messaging metrics provided by Cassandra from Java using JMX. I get the correct results when I use the following query with swiss java knife

java -jar sjk-plus-0.4.2.jar mx -s localhost:7100 -mg -all -b org.apache.cassandra.metrics:type=Messaging,name=* -f Mean
org.apache.cassandra.metrics:type=Messaging,name=CrossNodeLatency
1331.0469921040174
org.apache.cassandra.metrics:type=Messaging,name=datacenter1-Latency
1331.1071897694487

However, with the following Java code I get a javax.management.InstanceNotFoundException: org.apache.cassandra.metrics:type=Messaging exception.

JMXServiceURL url = null;
        try {
            url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://127.0.0.1:7100/jmxrmi");
        } catch (MalformedURLException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        JMXConnector mConnector = null;
        try {
            mConnector = JMXConnectorFactory.connect(url);
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        MBeanServerConnection mMBSC = null;
        try {
            mMBSC = mConnector.getMBeanServerConnection();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        ObjectName mObjectName = null;
        try {
            mObjectName = new ObjectName("org.apache.cassandra.metrics:type=Messaging");
        } catch (MalformedObjectNameException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        Set<ObjectName> myMbean = null;
        try {
            myMbean = mMBSC.queryNames(mObjectName, null);
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        try {
            System.out.println((mMBSC.getAttribute(mObjectName, "*")).toString());
        } catch (AttributeNotFoundException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        } catch (InstanceNotFoundException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        } catch (MBeanException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        } catch (ReflectionException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }

Could someone please explain where am I making a mistake?


Answer:

Swiss Java Knife seems to accept an ObjectName wildcard (or pattern) as:

org.apache.cassandra.metrics:type=Messaging,name=*

but your code is simply looking up a non-pattern MBean as:

org.apache.cassandra.metrics:type=Messaging

Change your code to:

  1. Use the pattern
  2. queryNames will return a Set of matching ObjectNames, so iterate through the set and query each one.
  3. You need to specify a string array of actual MBean attribute names, not "*" as in your code. (It would be nice if that was supported)

Something like:

mObjectName = new ObjectName("org.apache.cassandra.metrics:type=Messaging,name=*");
Set<ObjectName> names = mMBSC.queryNames(mObjectName, null);
for(ObjectName on: names) {
  System.out.println(on + "\n" + mMBSC.getAttribute(on, "Mean").toString());
}

Question:

I want to write a simple Java code to monitor a Cassandra database with JMX. Now I'm stuck with retrieving the CPU usage of the database. As far as I figured out, a possible MBean would be the java.lang:type=OperatingSystem with attribute ProcessCpuLoad.

However, it seems in this case the result would be the CPU usage of all processes running in the JVM, not only the Cassandra threads. Is this assumption correct?

I also wonder what data is shown as CPU usage when connecting with JConsole to the database. Is it possible to get direct access to these values(I mean without JConsole)? Or is there another Mbean which gives exactly the desired values?

Thanks, Nico


Answer:

ProcessCpuLoad in the OS mbean is correct. Its not all JVMs, just the one JVM thats reporting it. You do not have multiple processes running within a single JVM, the JVM runs as a single process per java application.

You can use java.lang:type=Threading to monitor cpu time spent on individual threads but there are a ton of threads in Cassandra and it will probably never be totally right (miss things like GC time).

If dont want to use jconsole you can check:

ps -p <whatever-your-cassandra-pid-is> -o %cpu

# or depending on OS/installer 

ps -p `cat /var/run/cassandra.pid` -o %cpu