Install and Configure a 3 node Cassandra Cluster on Ubuntu 14.04

Note: This article was written by our Cloud Support Engineer, Azhagiri Panneerselvam

We had to setup a 3 node Cassandra cluster for a customer recently and we ran into a few issues while setting up. I thought of sharing installation steps and how we troublesooted the issues.

Installation

We had to install Cassandra 2.1.12 and it needs open-jdk7 and open-jre7. Lets install them.

sudo apt-get install openjdk-7-jre  
sudo apt-get install openjdk-7-jdk  

Check version and set default Java (if you had other versions of Java installed already). If multiple versions are displayed, choose the desired version as default by choosing the number.

java -version  
sudo update-alternatives --config java  

Set Java environment variable

sudo vi /etc/environment  
JAVA_HOME="/usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java"  

And path

sudo vi /etc/profile

--At th end of the file, add following lines.

JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java  
PATH=$PATH:$HOME/bin:$JAVA_HOME/bin  
export JAVA_HOME  
export JRE_HOME  
export PATH  

Install Cassandra

Add repo to the sourcelist.

echo "deb http://www.apache.org/dist/cassandra/debian 21x main" | sudo tee -a /etc/apt/sources.list.d/cassandra.sources.list  

Add the keys. Execute the following commands one after the other.

gpg --keyserver pgp.mit.edu --recv-keys F758CE318D77295D  
gpg --export --armor F758CE318D77295D | sudo apt-key add -  
--add secondaru key
gpg --keyserver pgp.mit.edu --recv-keys 2B5C1B00  
gpg --export --armor 2B5C1B00 | sudo apt-key add -  
--add the third
gpg --keyserver pgp.mit.edu --recv-keys 0353B12C  
gpg --export --armor 0353B12C | sudo apt-key add -  

Update repository and install Cassandra

sudo apt-get update && sudo apt-get install cassandra  

Cluster Configuration

I am assuming that you installed Cassandra on all three nodes using steps above by now. My cluster looks like this -

Node Hostname Private IP
Node01 as01 10.1.1.4
Node02 as02 10.1.1.5
Node03 as03 10.1.1.6

The customer had a requirement to make all of these nodes as seeds. The seed node designation has no purpose other than bootstrapping the gossip process for new nodes joining the cluster. Seed nodes are not a single point of failure, nor do they have any other special purpose in cluster operations beyond the bootstrapping of nodes.

Stop Cassandra on each node - sudo service cassandra stop
Open the cassandra.yaml - sudo vi /etc/cassandra/cassandra.yaml

The following parameters must be change in all three nodes.

Node01

cluster_name: 'HeliconCluster'  
num_tokens: 256  
seed_provider:  
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "as01,as02,as03"

listen_address: 10.1.1.4  
broadcast_address: 10.1.1.4  
rpc_address: 0.0.0.0  
broadcast_rpc_address: 10.1.1.4  

Node02

cluster_name: 'HeliconCluster'  
num_tokens: 256  
seed_provider:  
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "as01,as02,as03"

listen_address: 10.1.1.5  
broadcast_address: 10.1.1.5  
rpc_address: 0.0.0.0  
broadcast_rpc_address: 10.1.1.5  

Node03

cluster_name: 'HeliconCluster'  
num_tokens: 256  
seed_provider:  
    - class_name: org.apache.cassandra.locator.SimpleSeedProvider
      parameters:
          - seeds: "as01,as02,as03"

listen_address: 10.1.1.6  
broadcast_address: 10.1.1.6  
rpc_address: 0.0.0.0  
broadcast_rpc_address: 10.1.1.6  

Start cassandra service - sudo service cassandra start

Before starting the other nodes you have to create keyspaces for replication. Get into the cassandra shell.

$ cqlsh
cqlsh> create keyspace Aurora WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : '3' };  
exit  

Check the cluster status - nodetool status aurora

After creating the keyspace in node01 then start the service in other two nodes. Once the cassandra starts, you should see image like below.

Validate the cluster

Connect to Node01 and open cqlsh - cqlsh use <keyspace name>
Create a table. Insert some records if you wish.

CREATE TABLE emp(  
   emp_id int PRIMARY KEY,
   emp_name text,
   emp_city text,
   emp_sal varint,
   emp_phone varint
   );
INSERT INTO emp (emp_id, emp_name, emp_city )  
  VALUES('xxxxxx', 'xxxxxx', 'xxxx' );

On Node02 and Node03 you should see this keyspace, table and the data you inserted.

Errors

While checking nodetool status, we saw an error stating it was not able to connect to any of the slaves.

Connection error: ('Unable to connect to any servers', {'127.0.0.1': error(111, "Tried connecting to [('127.0.0.1', 9042)]. Last error: Connection refused")})  

We checked syslog.

A node with address /10.1.2.5 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.  

Looks like a node already exists and it cant add a duplicate node. So to solve this, open the environments variable file and add the private ip of respective node.

sudo vi /etc/cassandra/cassandra-env.sh file 

JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=10.1.1.4"  

Similarly, another error - same solution as above.

ERROR [main] 2016-01-07 10:40:47,535 CassandraDaemon.java:579 - Exception encountered during startup  
java.lang.RuntimeException: Cannot replace_address /10.1.2.5 because it doesn't exist in gossip.  

Hope this was helpful. If you found this interesting, you might also like another Cassandra post from our blog - Cassandra Backup and Recovery using OpsCenter

Happy clustering! :)

comments powered by Disqus