Splunk NGINX search head load balancing

Sunday, January 12, 2020 » linux, nginx, splunk

This is a basic guide to set up nginx load balancing for splunk search head clustering using containers. The guide is running docker with root and is by no means a production worthy configuration.

Install docker.

Add the docker repo.

1
2
3
➜  ~ yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

Install docker

1
➜  ~ yum install docker-ce docker-ce-cli containerd.io docker-compose -y

Start docker.

1
➜  ~ systemctl enable docker ; systemctl start docker

Configure host.

Add 443 to firewall.

1
firewall-cmd service-add=https --permanent --zone=public

enable ip forwarding in /etc/sysctl.conf

1
net.ipv4.ip_forward=1

Configure docker.

The file structure for docker file.

1
2
3
4
5
docker
    \ load-balancer
        | Dockerfile
        | nginx.conf
    | docker-compose.yal

docker-compose.yml

1
2
3
4
5
6
7
8
9
version: "3.3"
services:
  loadbalancer:
    build: ./load-balancer
    tty: true
    ports:
      - '443:443'
volumes:
  backend:

Dockerfile

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
# Use the standard Nginx image from Docker Hub
FROM nginx
# The Dockerfile's author
LABEL Hawk Davis
# Copy the configuration file from the current directory and paste
# it inside the container to use it as Nginx's default config.
COPY nginx.conf /etc/nginx/nginx.conf
# Port 443 of the container will be exposed and then mapped to port
# 443 of our host machine via Compose. This way we'll be able to
# access the server via localhost:443 on our host.
EXPOSE 443

# Start Nginx when the container has provisioned.
CMD ["nginx", "-g", "daemon off;"]

Using TCP load balancing instead of HTTP load balancing for security reasons.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
events {
  worker_connections 1024;
}

stream {

  upstream splunk.example.com {
    hash $remote_addr;
    server 10.0.0.10:443;
    server 10.0.0.11:443;
    server 10.0.0.12:443;
  }

  server {
    listen 443;
    proxy_pass splunk.example.com;
  }
}

run container

To run the container, cd into the docker file and run the following.

1
docker-compose up --build

Expect improvements to this guide in the future.