Hot questions for Using Cassandra in ssl

Question:

I want to setup my own CA for use with a cassandra cluster so that I do not have to copy all of the certificates around every time I add a new node. I have read a few tutorials for Cassandra and SSL but they all work with copying certificates around. I am a little lost in the CA process

This is what I think I need to do

  • Create CA once
  • Create a CSR per node and then sign each with my CA (-> save as nodeX.crt)
  • import the node0.crt to my cassandra node0 keystore, node1.crt to node1 keystore etc

Now:

  • Do I need to add anything to the truststore?
  • Do I need to do anything with the CA file? Copy it to every client / node server?
  • What file do I need to provide my java client with? the cqlsh client?

Advantage: No more copying of ssl certificates between nodes. Just one for each node and you're set.

edit:

Ok this is how I did it. If I made any mistakes, please let me know. I left out things like JCE files and proper cassandra.yaml config. These need to be present on server!

openssl genrsa -out clusterCA.key 2048
openssl req -x509 -new -key clusterCA.key -days <DAYS> -out clusterCA.pem

keytool -importcert -alias clusterCA -file clusterCA.pem -keystore clustertruststore -storepass <PASS>

#on each cassandra host for clients. for client replace nodename with clientname
keytool -genkeypair -alias <NODENAME> -keyalg RSA -keysize 2048 -dname "CN=<NODENAME>,OU=<UNITNAME>,O=<ORGANISATION>" -keypass <PASS> -keystore <NODENAME>.keystore -storepass <PASS> -validity <DAYS>

keytool -keystore <NODENAME>.keystore -alias <NODENAME> -certreq -file <NODENAME>.cert -storepass <PASS> -keypass <PASS>


# sign it with CA

openssl x509 -req -CA clusterCA.pem -CAkey clusterCa.key -in <NODENAME>.cert -out <NODENAME>.signed -days <DAYS> -CAcreateserial

# add rootCA to host

keytool -keystore <NODENAME>.keystore -storepass <PASS> -alias clusterCA -import -file clusterCA.pem -noprompt

keytool -keystore <NODENAME>.keystore -storepass <PASS> -alias <NODENAME> -import -file <NODENAME>.signed -keypass <PASS>

## use <NODENAME>.keystore as truststore and keystore for cassandra node / client trust/keystore
## No need to copy keystores around. You only need it on your host


## create CQLSH pem
keytool -importkeystore -srckeystore <NODENAME>.keystore -destkeystore <NODENAME>_user1.p12 -deststoretype PKCS12
openssl pkcs12 -in <NODENAME>_user1.p12 -out <NODENAME>_user1.pem -nodes

##  use <NODENAME>_user1.pem as certfile for cqlsh

Answer:

Your strategy is very sound and that is the way I'd do it. You want to have your own Certificate Authority and then create a CSR for each node. That's much easier to manage than trusting node certificates individually.

  • Each node will have it's own keystore storing it's cert.
  • You will want every node to have the CAs public certificate in its truststore. This is only if you have 'require_client_auth' set to true. I'd recommend doing this as it isn't too difficult to set up and adds an extra layer of identification which should be considered important.

It is also important to differentiate between internode encryption and client encryption. Cassandra has different settings for each (documented in the links above). If using client-to-node encryption you will also want to have a truststore for client certificates. You could use the same trust store and also issue certificates to clients.

On the client-to-node side here's an example from the java-driver tests how to set up your SSLContext using your key and truststores:

/**
 * @param keyStorePath Path to keystore, if absent is not used.
 * @param trustStorePath Path to truststore, if absent is not used.
 * @return {@link com.datastax.driver.core.SSLOptions} with the given keystore and truststore path's for
 * server certificate validation and client certificate authentication.
 */
public SSLOptions getSSLOptions(Optional<String> keyStorePath, Optional<String> trustStorePath) throws Exception {

    TrustManagerFactory tmf = null;
    if(trustStorePath.isPresent()) {
        KeyStore ks = KeyStore.getInstance("JKS");
        ks.load(this.getClass().getResourceAsStream(trustStorePath.get()), DEFAULT_CLIENT_TRUSTSTORE_PASSWORD.toCharArray());

        tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
        tmf.init(ks);
    }

    KeyManagerFactory kmf = null;
    if(keyStorePath.isPresent()) {
        KeyStore ks = KeyStore.getInstance("JKS");
        ks.load(this.getClass().getResourceAsStream(keyStorePath.get()), DEFAULT_CLIENT_KEYSTORE_PASSWORD.toCharArray());

        kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());
        kmf.init(ks, DEFAULT_CLIENT_KEYSTORE_PASSWORD.toCharArray());
    }

    SSLContext sslContext = SSLContext.getInstance("TLS");
    sslContext.init(kmf != null ? kmf.getKeyManagers() : null, tmf != null ? tmf.getTrustManagers() : null, new SecureRandom());

    return new SSLOptions(sslContext, SSLOptions.DEFAULT_SSL_CIPHER_SUITES);
}

Once you an SSLOptions object you can simply pass it into your Cluster Builder, i.e.:

cluster = Cluster.builder()
    .addContactPoint(host)
    .withSSL(sslOptions))
    .build();

CQLSH supports SSL via the cqlshrc file. You can find an example of how to set that up here.

Question:

I am trying to enable client to node SSL encryption however I keep running into the most strangest errors when trying to start DSE after setting client_encryption_options as true in cassandra.yaml

this is from dse log :

ERROR 15:09:42,277  DseModule.java:108 - Unable to start server. Exiting...
com.google.inject.CreationException: Unable to create injector, see the following errors:

1) An exception was caught and reported. Message: Failed to initialize SSLContext: File '/home/ec2-user/keystore.node2' does not exist
  at com.datastax.bdp.DseModule.configure(Unknown Source)

1 error
        at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:466) ~[guice-4.0.jar:na]
        at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:155) ~[guice-4.0.jar:na]
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107) ~[guice-4.0.jar:na]
        at com.google.inject.Guice.createInjector(Guice.java:96) ~[guice-4.0.jar:na]
        at com.google.inject.Guice.createInjector(Guice.java:73) ~[guice-4.0.jar:na]
        at com.google.inject.Guice.createInjector(Guice.java:62) ~[guice-4.0.jar:na]
        at com.datastax.bdp.ioc.DseInjector.get(DseInjector.java:31) ~[dse-core-5.0.6.jar:5.0.6]
        at com.datastax.bdp.DseModule.main(DseModule.java:89) ~[dse-core-5.0.6.jar:5.0.6]
Caused by: org.apache.cassandra.exceptions.ConfigurationException: Failed to initialize SSLContext: File '/home/ec2-user/keystore.node2' does not exist
        at com.datastax.bdp.config.DseConfig.init(DseConfig.java:443) ~[dse-core-5.0.6.jar:5.0.6]
        at com.datastax.bdp.DseCoreModule.<init>(DseCoreModule.java:76) ~[dse-core-5.0.6.jar:5.0.6]
        at com.datastax.bdp.DseModule.getRequiredModules(DseModule.java:139) ~[dse-core-5.0.6.jar:5.0.6]
        at com.datastax.bdp.server.AbstractDseModule.configure(AbstractDseModule.java:27) ~[dse-core-5.0.6.jar:5.0.6]
        at com.datastax.bdp.DseModule.configure(DseModule.java:76) ~[dse-core-5.0.6.jar:5.0.6]
        at com.google.inject.AbstractModule.configure(AbstractModule.java:62) ~[guice-4.0.jar:na]
        at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:340) ~[guice-4.0.jar:na]
        at com.google.inject.spi.Elements.getElements(Elements.java:110) ~[guice-4.0.jar:na]
        at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:138) ~[guice-4.0.jar:na]
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104) ~[guice-4.0.jar:na]
        ... 5 common frames omitted
Caused by: java.io.FileNotFoundException: File '/home/ec2-user/keystore.node2' does not exist
        at org.apache.commons.io.FileUtils.openInputStream(FileUtils.java:299) ~[commons-io-2.4.jar:2.4]
        at org.apache.commons.io.FileUtils.readFileToByteArray(FileUtils.java:1763) ~[commons-io-2.4.jar:2.4]
        at com.datastax.bdp.util.SSLUtil.createKeyStore(SSLUtil.java:127) ~[dse-core-5.0.6.jar:5.0.6]
        at com.datastax.bdp.util.SSLUtil.initKeyManagerFactory(SSLUtil.java:115) ~[dse-core-5.0.6.jar:5.0.6]
        at com.datastax.bdp.config.DseConfig.resolveKeyManagerFactorySafely(DseConfig.java:831) ~[dse-core-5.0.6.jar:5.0.6]
        at com.datastax.bdp.config.DseConfig.getSSLContext(DseConfig.java:737) ~[dse-core-5.0.6.jar:5.0.6]
        at com.datastax.bdp.config.DseConfig.init(DseConfig.java:439) ~[dse-core-5.0.6.jar:5.0.6]
        ... 14 common frames omitted

This is my cassandra.yaml file

client_encryption_options:
    enabled: false
    # If enabled and optional is set to true encrypted and unencrypted connections are handled.
    optional: false
    keystore: /home/ec2-user/keystore.node2
    keystore_password: cassandra
    require_client_auth: true
    # Set trustore and truststore_password if require_client_auth is true
    truststore: /home/ec2-user/truststore.node2
    truststore_password: cassandra
    # More advanced defaults below:
    protocol: TLS
    algorithm: SunX509
    store_type: JKS
    cipher_suites: [TLS_RSA_WITH_AES_256_CBC_SHA]

I am not sure why I keep getting

SSLContext: File '/home/ec2-user/keystore.node2' does not exist

This is the location of my keystore -

[ec2-user@ip-172-18-51-175 ~]$ locate keystore.node2
/home/ec2-user/keystore.node2

What am i possibly doing wrong ?


Answer:

switch to your cassandra user, and do a cat /home/ec2-user/keystore.node2 and see if that user can see the file

also run: ls -lna /home/ec2-user/keystore.node2 and post here.

Question:

I have a 3 cassandra nodes (db-stats-01, db-stats-02, db-stats-03), which are all Up and Normal according to nodetool status.

The client is essentially doing this:

Builder builder = Cluster.builder();
builder.addContactPoint(node);
context = getSSLContext();
String[] cipherSuites = SSLOptions.DEFAULT_SSL_CIPHER_SUITES; 

builder.withSSL(new SSLOptions(context, cipherSuites)).build();

and failing on the last line with the following exception:

Sep 12 14:31:26 localhost daemon: Defuncting connection to db-stats-01/192.168.105.1
Sep 12 14:31:26 localhost com.datastax.driver.core.ConnectionException: [db-stats-01/192.168.105.1] Unexpected error during transport initialization (com.datastax.driver.core.ConnectionException: [db-stats-01/192.168.105.1] Operation Timeouted)
Sep 12 14:31:26 localhost     at com.datastax.driver.core.Connection.initializeTransport(Connection.java:182)
Sep 12 14:31:26 localhost     at com.datastax.driver.core.Connection.<init>(Connection.java:132)
Sep 12 14:31:26 localhost     at com.datastax.driver.core.Connection.<init>(Connection.java:60)
Sep 12 14:31:26 localhost     at com.datastax.driver.core.Connection$Factory.open(Connection.java:419)
Sep 12 14:31:26 localhost     at com.datastax.driver.core.ControlConnection.tryConnect(ControlConnection.java:205)
Sep 12 14:31:26 localhost     at com.datastax.driver.core.ControlConnection.reconnectInternal(ControlConnection.java:168)
Sep 12 14:31:26 localhost     at com.datastax.driver.core.ControlConnection.connect(ControlConnection.java:81)
Sep 12 14:31:26 localhost     at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:794)
Sep 12 14:31:26 localhost     at com.datastax.driver.core.Cluster$Manager.access$100(Cluster.java:721)
Sep 12 14:31:26 localhost     at com.datastax.driver.core.Cluster.<init>(Cluster.java:82)
Sep 12 14:31:26 localhost     at com.datastax.driver.core.Cluster.<init>(Cluster.java:67)

Sometimes, but not everytime, I get the following error on the server side:

    DEBUG [New I/O worker #1] 2014-09-12 14:36:14,337 Slf4JLogger.java (line 36) SSLEngine.closeInbound() raised an exception after a handshake failure.
javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1619)
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1587)
    at sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:1517)
    at org.jboss.netty.handler.ssl.SslHandler.setHandshakeFailure(SslHandler.java:1407)
    at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1293)
    at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:913)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
DEBUG [New I/O worker #1] 2014-09-12 14:36:14,342 Slf4JLogger.java (line 36) Swallowing an exception raised while writing non-app data
java.nio.channels.ClosedChannelException
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:434)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:129)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
    at org.jboss.netty.channel.Channels.write(Channels.java:725)
    at org.jboss.netty.channel.Channels.write(Channels.java:686)
    at org.jboss.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:1153)
    at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1246)
    at org.jboss.netty.handler.ssl.SslHandler.channelDisconnected(SslHandler.java:656)
    at org.jboss.netty.channel.Channels.fireChannelDisconnected(Channels.java:396)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:361)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

However, most of the time I don't even see this error on the server, dispite the server being set to DEBUG.

I don't suspect a connectivity issue, because I have watched tcpdump and I see packets going to and from the client and server. Also, there is briefly an ESTABLISHED connection up on port 9042 when I look at netstat -anp.

Using telnet, I can verify that the server is listening on 9042, and when I send junk through telnet I get a nice error on the server side complaining about a non-SSL record:

ERROR [Native-Transport-Requests:85] 2014-09-12 14:40:23,747 ErrorMessage.java (line 222) Unexpected exception during request
org.jboss.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 640d0a64660d0a

Not sure what else to check or where I may have gone wrong. Before implementing SSL in my client, I didn't see this issue at all, so my thought is it is probably related. Other misc info:

[root@db-stats-01 ~]# rpm -qa | grep cassandra
cassandra20-2.0.9-1.noarch
[root@db-stats-01 ~]# java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

Answer:

The SSL setup was wrong in some way. I regenerated all my key and trust stores according to http://techdocs.acunu.com.s3.amazonaws.com/v5.0/admin/security/ssl.html and everything worked as expected.

Question:

Cassandra is on version 3.11 and runs in docker

i have a problem with the following entry in cassandra.yml require_client_auth: true in the client_encryption_options block.

If I set to true I get the following error message in the log when I run cqlsh --ssl:

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: null cert chain
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:934) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:397) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:302) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) [netty-all-4.0.44.Final.jar:4.0.44.Final]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_232]

Certificate were generated using this page: https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/configuration/secureSSLCertWithCA.html

In the cqlshrc the following is written:

[authentication]
username = xxx
password = xxx

[connection]
hostname = 127.0.0.1
port = 9042

[ssl]
certfile = /etc/cassandra/xxx.crt
validate = false

I have regenerated and integrated the certificates several times. Without success.

Edit:

okay now the error disappears with the zero cert chain. When executing cqlsh --ssl the following will happen:

Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(0, u "Tried connecting to [('127.0.0.1', 9042)]. Last error: unknown error (_ssl.c:2947)")}) I am not sure which file to convert to a PKCS12 file. I have generated the following certificates:

node1.keystore, server-truststore, node1.csr, node1.crt_signed, rootCa.srl, rootCa.key and rootCa.crt

From which of the files should I generate the PKCS12 file?

Sorry I am new to the crypto world.

My .pem file looked like this now:

Bag Attributes
    friendlyName: node1
    localKeyID: 54 69 6D 65 20 31 35 38 35 30 36 39 35 35 35 37 33 36
subject=/C=US/O=YourCompany/OU=TestCluster/CN=node1
issuer=/C=US/O=YourCompany/OU=TestCluster/CN=rootCa
-----BEGIN CERTIFICATE-----
MIIDFTCCAf0CCQDSyTTMgIHuHDANBgkqhkiG9w0BAQUFADBKMQswCQYDVQQGEwJV
UzEUMBIGA1UECgwLWW91ckNvbXBhbnkxFDASBgNVBAsMC1Rlc3RDbHVzdGVyMQ8w
DQYDVQQDDAZyb290Q2EwHhcNMjAwMzI0MTY1NzMwWhcNMjEwMzI0MTY1NzMwWjBP
MQswCQYDVQQGEwJVUzEUMBIGA1UEChMLWW91ckNvbXBhbnkxFDASBgNVBAsTC1Rl
c3RDbHVzdGVyMRQwEgYDVQQDEwsxMC4yMC4wLjEwMTCCASIwDQYJKoZIhvcNAQEB
BQADggEPADCCAQoCggEBAMVTnpvTvMEFswzhTUeJYoTHB04MBx6hEh1pirqhlfVA
Nd8DJ6hR4KZFZ8nMxsw5etkR9iyPVaN4/Hdem6r/t8tHrPWPIADPULR9Kh7Y1s7f
dvAnro0ZpyMtLmabhbxEk4Z0B7ku8NkYdsCuQsIsCZEM0pGDYv3VOmyV/LryQA2x
qGYc0DNYXAVuaqlXrnFc/w4l9UQFxDwWDx05VB0wxXTXf/NnF2UZAB3FZIBIJQhf
pEqbdNiTK4nzK//RF+SBN+2o5YP12CIaDr+163t/2PAcm0W+attmDox5+41IQSF2
CTMW7XXjuIvJ6lLe6HfqouURWPtkyu86ik+p29kaEpMCAwEAATANBgkqhkiG9w0B
AQUFAAOCAQEAJgMDzdcE1CyjERjuxOuTxJYk9QSefLAJI4lFPOO0vNlbkteippE+
vWEc/AvYAEWVHUVPZbOvL45XkaIsUGV9pXpC4/XGv0eMp+sPyEcHdaALxc5n1qsj
ag2FhqVHCoyfwYIIefiS7nXKENKtBTMDSuNNmNMb/cksACP7rouz6ID74u7lDMBH
VcZpzu6ttpO+aLNBe3iSbDA1Ne8Nh33tm94L4FhE7oorA7o+urT1PlWARAopDtzk
Bj0DGK0mco4w1dJazAhDkii39XkAWfs3LvpXC1DijMC8O51iTJJho5bGGcPVe4dy
+9JId48wrN1sTcLhPyI60uEZGDGr8awKwQ==
-----END CERTIFICATE-----
Bag Attributes
    friendlyName: CN=rootCa,OU=TestCluster,O=YourCompany,C=US
subject=/C=US/O=YourCompany/OU=TestCluster/CN=rootCa
issuer=/C=US/O=YourCompany/OU=TestCluster/CN=rootCa
-----BEGIN CERTIFICATE-----
MIIDEDCCAfgCCQC64l3iiXV4SDANBgkqhkiG9w0BAQsFADBKMQswCQYDVQQGEwJV
UzEUMBIGA1UECgwLWW91ckNvbXBhbnkxFDASBgNVBAsMC1Rlc3RDbHVzdGVyMQ8w
DQYDVQQDDAZyb290Q2EwHhcNMjAwMzI0MTY1MzU2WhcNMjEwMzI0MTY1MzU2WjBK
MQswCQYDVQQGEwJVUzEUMBIGA1UECgwLWW91ckNvbXBhbnkxFDASBgNVBAsMC1Rl
c3RDbHVzdGVyMQ8wDQYDVQQDDAZyb290Q2EwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQDCC7NIswTCqQVoECBRuH8Z3HimZhLeLG+VWXJ6Um4TMB80MVgB
ToLfqSnGwS0yCB2G6tZWoN8joiz/3bbfYXLnVggj48z4ZMxykHmKwWcsRgScKPU9
BeZW2/trlplbIkOEy8J544Q9212d9r1jnMXCITPKVmi/GVCD+VBxqYbD9nLGdNwM
D4q+s2BrRIq+13U52ntA86rvI+Jgu4SNBp5yD089kmgnrbRJwUvtCLZYc87u0rp9
5KI+5iag0A7v0Xu2/Dh3fQMO+YQrsajOo9OZ/dxhZBxdArWdgMqmC5R9kvPx3ypv
yM0rGaIrobj5y9QSm77ptraO02wJR4I/USsFAgMBAAEwDQYJKoZIhvcNAQELBQAD
ggEBAE+Bm7q05PdW3UZJ6ozl72/zuMDUQxDIiGQO+caNNyyqPy/d6A0+cwJ5J0JS
eQeS2Cg8knk5cjIsB5Ah3TuMnWPaOS7Eb8PrKjN7+up9ZxMMuIrLIUllYDFzfUMf
rMJ/mxR+8whKq9Umec3pAtGfzRwFdNJn9gqis71/02gK10cmq9MN0tyi7hD+9kTE
iC/mqJX5iCyiFluwRxlVP1tAPaD4RRtafpFHX9X9FYksUf/8mwr+3pNwfdBfIXW7
8A+k198jx326SbtQZBA5yWPTKpI9Mamm0704NjA2OH/0OceBq2rYv7GKfRYQxC4/
DUAlJYlfxCyqxjGqBp/RFMJ1Epg=
-----END CERTIFICATE-----
Bag Attributes
    friendlyName: rootca
    2.16.840.1.113894.746875.1.1: <Unsupported tag 6>
subject=/C=US/O=YourCompany/OU=TestCluster/CN=rootCa
issuer=/C=US/O=YourCompany/OU=TestCluster/CN=rootCa
-----BEGIN CERTIFICATE-----
MIIDEDCCAfgCCQC64l3iiXV4SDANBgkqhkiG9w0BAQsFADBKMQswCQYDVQQGEwJV
UzEUMBIGA1UECgwLWW91ckNvbXBhbnkxFDASBgNVBAsMC1Rlc3RDbHVzdGVyMQ8w
DQYDVQQDDAZyb290Q2EwHhcNMjAwMzI0MTY1MzU2WhcNMjEwMzI0MTY1MzU2WjBK
MQswCQYDVQQGEwJVUzEUMBIGA1UECgwLWW91ckNvbXBhbnkxFDASBgNVBAsMC1Rl
c3RDbHVzdGVyMQ8wDQYDVQQDDAZyb290Q2EwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQDCC7NIswTCqQVoECBRuH8Z3HimZhLeLG+VWXJ6Um4TMB80MVgB
ToLfqSnGwS0yCB2G6tZWoN8joiz/3bbfYXLnVggj48z4ZMxykHmKwWcsRgScKPU9
BeZW2/trlplbIkOEy8J544Q9212d9r1jnMXCITPKVmi/GVCD+VBxqYbD9nLGdNwM
D4q+s2BrRIq+13U52ntA86rvI+Jgu4SNBp5yD089kmgnrbRJwUvtCLZYc87u0rp9
5KI+5iag0A7v0Xu2/Dh3fQMO+YQrsajOo9OZ/dxhZBxdArWdgMqmC5R9kvPx3ypv
yM0rGaIrobj5y9QSm77ptraO02wJR4I/USsFAgMBAAEwDQYJKoZIhvcNAQELBQAD
ggEBAE+Bm7q05PdW3UZJ6ozl72/zuMDUQxDIiGQO+caNNyyqPy/d6A0+cwJ5J0JS
eQeS2Cg8knk5cjIsB5Ah3TuMnWPaOS7Eb8PrKjN7+up9ZxMMuIrLIUllYDFzfUMf
rMJ/mxR+8whKq9Umec3pAtGfzRwFdNJn9gqis71/02gK10cmq9MN0tyi7hD+9kTE
iC/mqJX5iCyiFluwRxlVP1tAPaD4RRtafpFHX9X9FYksUf/8mwr+3pNwfdBfIXW7
8A+k198jx326SbtQZBA5yWPTKpI9Mamm0704NjA2OH/0OceBq2rYv7GKfRYQxC4/
DUAlJYlfxCyqxjGqBp/RFMJ1Epg=
-----END CERTIFICATE-----

with this .pem file i get this null cert chain error


Answer:

If you use CA signed certificate, and generate the cert files by following the https://docs.datastax.com/en/cassandra-oss/3.0/cassandra/configuration/secureSSLCertWithCA.html, the cert file used in cqlshrc is the rootCa.crt.

Here is an example:

[connection]
port = 9042
factory = cqlshlib.ssl.ssl_transport_factory

[ssl]
certfile = /var/tmp/rootCa.crt
validate = true
userkey = /var/tmp/rootCa.key  ;<<<<< optional for 2 way SSL only
usercert = /var/tmp/rootCa.crt  ;<<<< optional for 2 way SSL only

Question:

I am trying to connect to cassandra cluster via java datastax driver with openssl as per this doc https://docs.datastax.com/en/developer/java-driver/3.1/manual/ssl/ with my client cert and key and the truststore as my cassandra cluster requires two way mutual cert authentication

Here is my code

public static void main( String[] args ) throws Exception
    {
        KeyStore ks = KeyStore.getInstance("JKS");
        // make sure you close this stream properly (not shown here for brevity)
        InputStream trustStore = new FileInputStream("MyTrustStore");
        ks.load(trustStore, "abcdef".toCharArray());
        TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
        tmf.init(ks);

        SslContextBuilder builder = SslContextBuilder
          .forClient()
          .sslProvider(SslProvider.OPENSSL)
          .trustManager(tmf)
          // only if you use client authentication
          .keyManager(new File("client_cert"), new File("private_key"));


        SSLOptions sslOptions = new NettySSLOptions(builder.build());


        Cluster cluster = Cluster.builder()
          .addContactPoint("w.x.y.z")
          .withSSL(sslOptions)
          .build();


    }

with the following dependencies in my pom

<dependencies>
        <dependency>
            <groupId>com.datastax.cassandra</groupId>
            <artifactId>cassandra-driver-core</artifactId>
            <version>3.6.0</version>
        </dependency>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-tcnative</artifactId>
            <version>2.0.25.Final</version>
        </dependency>

        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-tcnative-boringssl-static</artifactId>
            <version>2.0.0.Final</version>
            <classifier>osx-x86_64</classifier>
        </dependency>
</dependencies>

But I get an error as

Exception in thread "main" java.lang.UnsatisfiedLinkError: failed to load the required native library
    at io.netty.handler.ssl.OpenSsl.ensureAvailability(OpenSsl.java:327)
    at io.netty.handler.ssl.ReferenceCountedOpenSslContext.<init>(ReferenceCountedOpenSslContext.java:193)
    at io.netty.handler.ssl.ReferenceCountedOpenSslContext.<init>(ReferenceCountedOpenSslContext.java:182)
    at io.netty.handler.ssl.OpenSslContext.<init>(OpenSslContext.java:34)
    at io.netty.handler.ssl.OpenSslClientContext.<init>(OpenSslClientContext.java:188)
    at io.netty.handler.ssl.SslContext.newClientContextInternal(SslContext.java:775)
    at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:446)
    at com.example.App.main(App.java:41)
Caused by: java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_osx_x86_64, netty_tcnative_x86_64, netty_tcnative]
    at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:93)
    at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:421)
    at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:89)
    ... 7 more

I have tried removing the boringssl-static dep or the tcnative dep from pom but it still doesnt seem to work. Any help would be greatly appreciated.

Thank you in advance


Answer:

I suspect what could possibly be happening here is caused by some incompatibility between the versions of netty, netty-tcnative and netty-tcnative-boringssl-static being used here.

In previous experimentation, I have found that versions of netty and netty-tcnative are particularly important as there can be incompatibilities between the two.

datastax java driver 3.6.0 depends on netty 4.0.56.Final and lists netty-tcnative 2.0.7.Final as an optional dependency. You can also find in the documentation for java driver 3.6.0 that 2.0.7.Final is recommended:

There are known runtime incompatibilities between newer versions of netty-tcnative and the version of netty that the driver uses. For best results, use version 2.0.7.Final.

I also suspect that not using the same version of netty-tcnative and netty-tcnative-boringssl-static might cause incompatibilities. I would recommend trying the same versions.

Since I've tested this, I would first try the following configuration:

<dependencies>
    <dependency>
        <groupId>com.datastax.cassandra</groupId>
        <artifactId>cassandra-driver-core</artifactId>
        <version>3.6.0</version>
    </dependency>
    <dependency>
        <groupId>io.netty</groupId>
        <artifactId>netty-tcnative</artifactId>
        <version>2.0.7.Final</version>
    </dependency>
    <dependency>
        <groupId>io.netty</groupId>
        <artifactId>netty-tcnative-boringssl-static</artifactId>
        <version>2.0.7.Final</version>
        <classifier>osx-x86_64</classifier>
    </dependency>
</dependencies>