Hot questions for Using Cassandra in jvm

Question:

How to set Cassandra (>2.0) JVM heap size of 8GB? When I type in free -m it gives me the following. How can I set the Cassandra JVM heap size to 8GB?

             total       used       free     shared    buffers     cached
Mem:         16047      11336       4711          0         81       5814
-/+ buffers/cache:       5441      10606
Swap:            0          0          0

Answer:

As stated in the Tuning Java resources section of the documentation:

If you decide to change the Java heap sizing, both MAX_HEAP_SIZE and HEAP_NEWSIZE should should be set together in conf/cassandra-env.sh.

Inside cassandra-env.sh, search for this line:

#MAX_HEAP_SIZE="4G"

That line in the code happens after the MAX_HEAP_SIZE calculations have happened, so it works as an override. Uncomment it, and set it accordingly.

MAX_HEAP_SIZE="8G"

Be careful going above 8G. For guidance on tuning the JVM, I recommend Amy Tobey's Cassandra 2.1 Tuning Guide. She offers some tips for tuning CMS, as well as a great section on implementing the new G1GC.

Question:

I will be hosting my Cassandra database on Google cloud. Instances are priced in a linear fashion meaning 1cpu with 2gb ram is $1, 2cpu with 4gb is $2, 4cpu with 8GB is $4 and so on.

I am deciding on the size of my instances and am not sure what the standard is? I was thinking of using more fewer larger instances (8cpu, 64gb) opposed to lighter such as (2cpu, 4 gb). My thought process is with more instances each node will carry less of the overall data which would have a smaller impact if nodes fail. As well, the os of these smaller instances would have less overhead because it would accept less connections.

These are pros, but here are some cons I can think of: 1) Each instance will be less utilized 2) Cassandra + JVM overhead on so many instances can add up and be a lot of overhead. 3) I will be using local SSD opposed to persistent SSD which are much more expensive meaning each instance will need their own local SSD which raises costs.

These are some reasons I can think of, is there any other pros/cons between choosing more smaller instances vs fewer larger for a Cassandra database (maybe even nodes in general)? Are there any best practices associated to choosing Cassandra server sizes?

PS: I added the 'Java' tag because Cassandra is built using JAVA and runs on the JVM and would like to see if the JVM has any pros/cons.


Answer:

I think you've hit some of the tradeoff points, but here are a few other things:

  1. As the amount of data stored on a single node increases, the cost of bootstrapping (adding new nodes) increases. For instance, you'll get reasonable bootstrapping times storing 100 GB per node, but the process will take eons with 10 TB per node.
  2. SSD usage makes this less important, but consider using separate physical disks for your commitlog and data.
  3. Configurations with fewer than 4 cores or less than 8 GB of memory are usually not recommended, but your mileage may vary.

Question:

I am getting this error, trying to launch Cassandra. I ran rm -rf * on /var/log/cassandra and /var/lib/cassandra, and try to run cassandra again, with no success. Any idea? I have been looking to similar cases, but not at the launch of Cassandra, and nothing I found helped me to solve this problem.

root@test # cassandra
(...)
16:38:38.551 [MemtableFlushWriter:1] ERROR o.a.c.service.CassandraDaemon - Exception in thread Thread[MemtableFlushWriter:1,5,main]
java.lang.RuntimeException: Insufficient disk space to write 572 bytes
        at org.apache.cassandra.db.Directories.getWriteableLocation(Directories.java:349) ~[apache-cassandra-2.2.7.jar:2.2.7]
        at org.apache.cassandra.db.Memtable.flush(Memtable.java:324) ~[apache-cassandra-2.2.7.jar:2.2.7]
        at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1165) ~[apache-cassandra-2.2.7.jar:2.2.7]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_65]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_65]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65]
16:38:41.785 [ScheduledTasks:1] INFO  o.a.cassandra.locator.TokenMetadata - Updating topology for all endpoints that have changed

FYI, if I try to run cassandra again just after, I got

[main] ERROR o.a.c.service.CassandraDaemon - Port already in use: 7199; nested exception is:
        java.net.BindException: Address already in use

So it seems that "CassandraDaemon" is alive; but if I want to run cqlsh I got this error:

root@test # cqlsh
Warning: custom timestamp format specified in cqlshrc, but local timezone could not be detected.
Either install Python 'tzlocal' module for auto-detection or specify client timezone in your cqlshrc.

Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})

Finally, the free -m command gives me

              total        used        free      shared  buff/cache   available
Mem:          15758        8308         950        1440        6499        5576
Swap:          4095        2135        1960

Thanks for you help!

EDIT : Here are the WARN message I got during Cassandra launch:

11:36:14.622 [main] WARN  o.a.c.config.DatabaseDescriptor - Small commitlog volume detected at /var/lib/cassandra/commitlog; setting commitlog_total_space_in_mb to 2487.  You can override this in cassandra.yaml
11:36:14.623 [main] WARN  o.a.c.config.DatabaseDescriptor - Only 17 MB free across all data volumes. Consider adding more capacity to your cluster or removing obsolete snapshots
11:36:14.943 [main] WARN  o.a.cassandra.service.StartupChecks - jemalloc shared library could not be preloaded to speed up memory allocations
11:36:14.943 [main] WARN  o.a.cassandra.service.StartupChecks - JMX is not enabled to receive remote connections. Please see cassandra-env.sh for more info.
11:36:14.943 [main] WARN  o.a.cassandra.service.StartupChecks - OpenJDK is not recommended. Please upgrade to the newest Oracle Java release
11:36:14.954 [main] WARN  o.a.cassandra.utils.SigarLibrary - Cassandra server running in degraded mode. Is swap disabled? : false,  Address space adequate? : true,  nofile limit adequate? : false, nproc limit adequate? : true

Answer:

As I said in a previous comment, it seems that the issue was triggered by the /var folder reaching its limit size, because Nifi, that I used for ingesting data, generates big logs in my configuration. Therefore removing useless logs and keeping an eye on the /var folder seems to prevent the error.