Elastic-Stack is a distributed and horizontally scalable search, analytics and visualization platform based on the components Elasticsearch, Logstash, Kibana and Beats. I’m quite overwhelmed with the options and integrations an Elastic-Stack has to offer. Switching from traditional log monitoring to Elastic-Stack is like switching from a horse-drawn carriage to the Falcon 9 rocket. Therefore this journey to Elastic-Stack will be a series of blog posts.
The goal of this blog post is a running Elasticsearch cluster with four nodes, a working Kibana visualization plus Nginx as reverse proxy on the first node and some Beats (Metricbeat, Filebeat) to monitor the cluster itself. The architecture of the cluster will look in this stage (Initial setup) like this:
Knowledge and skills in those topics might be helpful:
- Linux (Debian, CentOS, …)
- Logging (Log files, Syslog, Netflow, …)
- Network (IP addressing, Proxying, TCP/UDP ports, troubleshooting, …)
- Packet analyzer (tshark for proofing of TLS encryption, …)
- TLS (Certificates and Certificate Authority, …)
- YAML (YAML Ain’t Markup Language, configuration files, …)
Preparing the virtual machines
My home network/lab environment has two physical servers using XCP-NG as virtualization platform. This means a creation of four virtual machines acting as Elastic-Stack nodes and distributing two nodes per physical server is possible. For my small home network/lab environment the virtual machines will get the following resource assignment:
- 4 vCPU (4 socket / 1 core per socket)
- 8 GiB RAM (fixed allocation)
- 500 GiB Disk (on shared storage)
Preparing the Operating System
As Operating system for the Elastic-Stack nodes I’m using Debian 11 Bullseye with the latest netinstISO image. After a “standard” installation (Debian minimal plus SSH server) some additional, useful packages are installed. Those packages are:
- gnupg (used to install elastic’s key to apt key-ring)
- net-tools (to have netstat to check TCP/UDP ports)
- tshark (to verify if elastic traffic is encrypted)
As one-liner for copy&paste:
Because the virtual machines are all running on XCP-NG, the guest-tools will be installed as well. Detailed installation instructions are available in the documentation at XCP-NG guest-tools.
Installation of Elasticsearch
For the installation of Elasticsearch on Debian 11 I’m following the instructions provided at Install Elasticsearch with Debian Package. The documentation covers all aspects of setting up and configuring Elasticsearch and therefore I’m only referencing to it. Please note: The version at date of publishing this blog post is 7.15.
Configuration of Elasticsearch and Cluster bootstrapping (First node)
The first node is used to boot strap the cluster and therefore this node requires the configuration settings cluster.inital_master_node. Because Security and Encrypted communications are now included in the free and open-basic license I’m enabling the xpack feature in the YAML configuration file. For the licensing details included in free and open-basic, see Elastic Stack subscriptions.
To secure the cluster, the following instructions are provided and should (better: must) be set for production environments:
- Set up minimal security for Elasticsearch
- Set up basic security for the Elastic Stack
- Set up basic security for the Elastic Stack plus secured HTTPS traffic
The /etc/elasticsearch/elasticsearch.yml configuration file for the First node and cluster bootstrapping might look like this:
The first node of the Elastic-Stack cluster is then started with this command line:
Configuration of Elasticsearch (Second / Third / Fourth node)
A possible /etc/elasticsearch/elasticsearch.yml configuration file for the Second / Third / Fourth node might look like this:
After a start with systemctl start elasticsearch the Second, Third and Fourth node will automatically join the cluster due to the Cluster Discovery Seeding. To ensure the communication between Elasticsearch is encrypted I quick-check the communication with tshark:
The output of tshark should be similar like this:
Configuration of Kibana (First node)
The First node will also act as Kibana Dashboard for the data visualization. To install Kibana I’m following the standard installation process as described in Install Kibana with Debian package.
Because I’m operating my own Public key infrastructure for my home network/lab environment I can create inside the PKI a Certificate signing request and sign it directly with the Root certificate. The Certificate is then exported as PEM (Privacy-Enhanced Mail) format and transferred by SCP to the First node.
A possible /etc/kibana/kibana.yml configuration file might look like this:
Configuration of Nginx as Reverse Proxy (First node)
Because Kibana is only reachable via localhost (127.0.0.1) I set up Nginx to act as a reverse proxy server between a web browser and Kibana. Nginx can be installed on Debian with the following command line:
Because no other web site will be hosted on this server, the configuration can be done directly in the /etc/nginx/sites-available/default file. The configuration includes a certificate signed by my own PKI and acts as a Reverse Proxy for Kibana (running on 127.0.0.1:5601). A configuration file for Nginx might look like this:
When the configuration is completed, Nginx can be restarted with:
A quick test in a web browser should show the Elastic Welcome web page:
Add GeoIP for Metricbeat with Kibana Dev Tools
To add GeoIP information like Country, City, etc. per IP addresss to Metricbeat, the Elastic-Stack must be able to connect to the public Elastic GeoIP endpoint (See GeoIP processor). With the Kibana Dev Tools a pipeline for GeoIP can be added (See Enrich events with geoIP information). The adding of the pipeline looks similar like this:
Configuration of Metricbeat (First node)
To monitor the Elastic-Stack by itself, Metricbeat must have the modules beat-xpack, elasticsearch-xpack and system enabled. Because on the First node Kibana is installed, Kibana is monitored by Metricbeat as well. To activate the modules, the following command lines can be used:
A possible /etc/metricbeat/metricbeat.yml might look like the one shown below. Please note that this configuration is enriched with service tags to reflect the installed Metricbeat modules. The First node is also used to setup the Metricbeat dashboards in Kibana and requires therefore in the TLS configuration the Certificate of Kibana. Because the GeoIp pipeline was set up in Kibana Dev Tools, the pipeline can now be used by Metricbeat. The Elastic-Stack is monitoring itself (I’m not using a dedicated monitoring cluster) and therefore the monitoring.cluster_uuid must be set to itself.
Metricbeat module “system.yml” (First node)
A detailed description of this module is available at System module. The default configuration for the system.yml does not need any specific adjustments and therefore it might look like this:
Metricbeat module “beat-xpack.yml” (First node)
A detailed description of this module is available at Beat module. I just make sure that xpack.enabled is set to true and that metricsets are removed as suggested in the documentation:
Metricbeat module “elasticsearch-xpack.yml” (First node)
A detailed description of this module is available at Elasticsearch module. In this module I set the hosts configuration to all four Elasticsearch nodes and add the certificate of the ELasticsearch CA in certificate_authorities. A possible configuration file might look like this:
Metricbeat module “kibana-xpack.yml” (First node)
A detailed description of this module is available at Kibana module. Because Kibana is only reachable through localhost (127.0.0.1) and with TLS encryption, the configuration file must be adjusted with settings for hosts and the host’s TLS certificate. A possible configuration might look like shown below:
To setup the Metricbeat assets in Kibana I’m using the following command line:
Configuration of Metricbeat (Second / Third / Fourth node)
The configuration of Metricbeat for the Second, Third and Fourth node are similar to the ones described in Configuration of Metricbeat (First node), Metricbeat module “system.yml” (First node), Metricbeat module “beat-xpack.yml” (First node) and Metricbeat module “elasticsearch-xpack.yml” (First node) except for the kibana-xpack module. Please note: Because those nodes are not used to load Metricbeat dashboards into Kibana, the configuration part for Kibana Dashboards is removed.
After a restart of Metricbeat with systemctl restart metricbeat the Cluster monitoring should show in the overview that Elasticsearch and Kibana is monitored with Metricbeat-monitoring as shown below:
The screenshot shows that no Log data is found and for Log data Filebeat needs to be configured. This configuration will be done in one of my next blog posts.