Build a Simple Splunk Indexer Cluster

Disclaimer: This guide is to show a very simple step by step walk through of how to install and configure a Splunk Indexer Cluster. In my lab environment I am using the DVD install of CentOS 7, and the latest (at the time of install) version of Splunk which is version 7.2.3. There may be errors throughout this, there may also be better methods of completing an install. That being said, this guide has worked for me and should work for you!

I use acronyms a bit, if you are unfamiliar with them here’s a few that are common in the Splunk community:

  • HF – Heavy Forwarder
  • CM – Cluster Master
  • SH – Searchhead
  • IDX – Indexer
  • UF – Universal Forwarder
  • DS – Deployment Server
  • SHD/SH-D – Search Head Deployer

If you get stuck, join our discord and we may be able to point you in the right direction.

Let’s get started.

Install Splunk

On All machines (SH + CM + IDX’s):

  1. Use the latest splunk rpm:
wget -O splunk-7.2.3-06d57c595b80-linux-2.6-x86_64.rpm 'https://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=7.2.3&product=splunk&filename=splunk-7.2.3-06d57c595b80-linux-2.6-x86_64.rpm&wget=true'
  1. Install on default location “/opt/splunk”
rpm -i splunk-7.2.3-06d57c595b80-linux-2.6-x86_64.rpm
  1. Chown the /opt/splunk directory to splunk:splunk:
chown -R splunk:splunk /opt/splunk
  1. Before you start splunk, su to splunk (this user is created when installing with an rpm)
su splunk
  1. Now start splunk and accept the license (you’ll be prompted for an admin user & creds. We’ll use admin // password for this guide):
/opt/splunk/bin/splunk start --accept-license
  1. Return to the root user and enable Splunk to start on boot
exit
/opt/splunk/bin/splunk enable boot-start

Splunk is now installed with default settings.

Firewall

We’ll now configure the firewall on the CM for common ports (web / management / etc). We’ll also include our replication port of 9887.

sudo firewall-cmd --zone=public --add-port=8000/tcp --permanent
sudo firewall-cmd --zone=public --add-port=8089/tcp --permanent
sudo firewall-cmd --zone=public --add-port=9887/tcp --permanent
sudo firewall-cmd --reload

On the IDXs you’ll want to add your listening port as well (default 9997)

sudo firewall-cmd --zone=public --add-port=9997/tcp --permanent
sudo firewall-cmd --zone=public --add-port=8000/tcp --permanent
sudo firewall-cmd --zone=public --add-port=8089/tcp --permanent
sudo firewall-cmd --zone=public --add-port=9887/tcp --permanent
sudo firewall-cmd --reload

Build The Cluster

We’ll use the CLI for this portion. There are alternate options (via web GUI OR server.conf).

First thing to determine when building an indexer cluster is replication factor and search factor. In short replication factor is the number you set is a math equation:

(replication factor – 1) =  indexers you can tolerate losing

Let’s break that out a bit as it may be confusing. If you have two indexers with a replication factor of 2, you can lose 1 indexer but no more before it becomes an issue. In a larger environment with 10 indexers and a replication factor of 5 you can lose 4 indexers before bad things happen.

Search factor is similar, but it just determines how/what data is stored. Basically search factor dictates the searchable buckets stored. This can be the same as replication factor, but if it is less just know that search performance will degrade if you lose more indexers than the above equation can tolerate.

For more information on Replication factors and search factors I would suggest reading this post on answers.splunk.com: https://answers.splunk.com/answers/680401/replication-factor-with-n1-indexer.html

In my current lab we have 2 indexers. So we are going to do a replication factor of 2 with a search factor of 2.

Enable the Master

I’m going to use the IP’s for the machines, you can use the FQDN. Whatever you end up using don’t mix and match. If you use IP’s on one box, don’t use the FQDN on another. Also remember what you use for your cluster label. For the purpose of this guide I will literally call my label “cluster1” as shown below as well as use “your_key” as my literal secret key. Please don’t do this in a real environment.

su to splunk user and enable the master

su splunk
/opt/splunk/bin/splunk edit cluster-config -mode master -replication_factor 2 -search_factor 2 -secret your_key -cluster_label cluster1
  1. Enter your admin credentials as defined during install of splunk
  2. Exit splunk user & Restart splunk (This step might take some time!)
/opt/splunk/bin/splunk restart

Enable the Peer Nodes

  1. su to splunk and tell each indexer it belongs to a cluster:
su splunk
/opt/splunk/bin/splunk edit cluster-config -mode slave -master_uri https://cmFQDN_or_IP:8089 -replication_port 9887 -secret your_key
  1. Enter your admin credentials as defined during install of splunk
  2. Exit splunk user & Restart splunk (This step might take some time!)
/opt/splunk/bin/splunk restart

Now let’s logon to the CM and see if our cluster is happy. http://IP_of_CM:8000

Go to Settings > Indexer clustering

If your install worked correctly you’ll see a working cluster as shown below:

Connect Search Head to Indexer Cluster

Very much like the steps above, this just permits the search head to search the indexers.

  1. su to splunk and tell tell your searchhead(s) what indexers to search:
su splunk
splunk edit cluster-config -mode searchhead -master_uri https://cmFQDN_or_IP:8089 -secret your_key
  1. Enter your admin credentials as defined during install of splunk
  2. Exit splunk user & Restart splunk (This step might take some time!)
/opt/splunk/bin/splunk restart

Just like the previous step once the restart takes place go to the Cluster Master and see if our cluster is happy. http://IP_of_CM:8000 then Settings > Indexer clustering. You should now see a search head!

That’s it. If all went well you should now have a successful Splunk indexer cluster. Again if you get stuck, join us on discord.

Advertisements
One comment

Add Comment

Required fields are marked *. Your email address will not be published.