thierolf.org - Vom Hirn ins Terminal seit 1998.

Elastic-Stack for network engineers (Initial setup)

Written by Stefan on . Posted in Blog. Tags: #elastic-stack  #linux  #software 

Elastic-Stack is a distributed and horizontally scalable search, analytics and visualization platform based on the components Elasticsearch, Logstash, Kibana and Beats. I’m quite overwhelmed with the options and integrations an Elastic-Stack has to offer. Switching from traditional log monitoring to Elastic-Stack is like switching from a horse-drawn carriage to the Falcon 9 rocket. Therefore this journey to Elastic-Stack will be a series of blog posts.

The goal of this blog post is a running Elasticsearch cluster with four nodes, a working Kibana visualization plus Nginx as reverse proxy on the first node and some Beats (Metricbeat, Filebeat) to monitor the cluster itself. The architecture of the cluster will look in this stage (Initial setup) like this:

Elastic-Stack: Initial setup

Knowledge and skills in those topics might be helpful:

Preparing the virtual machines

My home network/lab environment has two physical servers using XCP-NG as virtualization platform. This means a creation of four virtual machines acting as Elastic-Stack nodes and distributing two nodes per physical server is possible. For my small home network/lab environment the virtual machines will get the following resource assignment:

Elastic-Stack: Virtual machine resource assignments

Preparing the Operating System

As Operating system for the Elastic-Stack nodes I’m using Debian 11 Bullseye with the latest netinstISO image. After a “standard” installation (Debian minimal plus SSH server) some additional, useful packages are installed. Those packages are:

As one-liner for copy&paste:


    apt -y install gnupg net-tools tshark

Because the virtual machines are all running on XCP-NG, the guest-tools will be installed as well. Detailed installation instructions are available in the documentation at XCP-NG guest-tools.

Installation of Elasticsearch

For the installation of Elasticsearch on Debian 11 I’m following the instructions provided at Install Elasticsearch with Debian Package. The documentation covers all aspects of setting up and configuring Elasticsearch and therefore I’m only referencing to it. Please note: The version at date of publishing this blog post is 7.15.

Configuration of Elasticsearch and Cluster bootstrapping (First node)

The first node is used to boot strap the cluster and therefore this node requires the configuration settings cluster.inital_master_node. Because Security and Encrypted communications are now included in the free and open-basic license I’m enabling the xpack feature in the YAML configuration file. For the licensing details included in free and open-basic, see Elastic Stack subscriptions.

To secure the cluster, the following instructions are provided and should (better: must) be set for production environments:

The /etc/elasticsearch/elasticsearch.yml configuration file for the First node and cluster bootstrapping might look like this:


    # ======================== Elasticsearch Configuration =========================
    cluster.name: "MY_CLUSTER-production"
    node.name: "HOST_1.TLD"
    node.attr.rack: "RACK_LOCATION"

    path.data: /var/lib/elasticsearch
    path.logs: /var/log/elasticsearch

    network.host: IP_ADDR_1
    http.port: 9200

    discovery.seed_hosts: ["IP_ADDR_1:9300", "IP_ADDR_2:9300", "IP_ADDR_3:9300", "IP_ADDR_4:9300"]

    # Required for Cluster Bootstrap only
    # Comment out after Bootstrap
    cluster.initial_master_nodes: ["HOST_1.TLD"]

    # Set Xpack to basic-license
    xpack.license.self_generated.type: basic
    xpack.monitoring.collection.enabled: true

    # Set Xpack security settings
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.client_authentication: required
    xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: http.p12
    xpack.security.authc.token.enabled: true
    xpack.security.authc.api_key.enabled: true

The first node of the Elastic-Stack cluster is then started with this command line:


    systemctl start elasticsearch

Configuration of Elasticsearch (Second / Third / Fourth node)

A possible /etc/elasticsearch/elasticsearch.yml configuration file for the Second / Third / Fourth node might look like this:


    # ======================== Elasticsearch Configuration =========================
    cluster.name: "MY_CLUSTER-production"
    node.name: "HOST_[2|3|4].TLD"
    node.attr.rack: "RACK_LOCATION"

    path.data: /var/lib/elasticsearch
    path.logs: /var/log/elasticsearch

    network.host: IP_ADDR_[2|3|4]
    http.port: 9200

    discovery.seed_hosts: ["IP_ADDR_1:9300", "IP_ADDR_2:9300", "IP_ADDR_3:9300", "IP_ADDR_4:9300"]

    # Set Xpack to basic-license
    xpack.license.self_generated.type: basic
    xpack.monitoring.collection.enabled: true

    # Set Xpack security settings
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.client_authentication: required
    xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
    xpack.security.http.ssl.enabled: true
    xpack.security.http.ssl.keystore.path: http.p12
    xpack.security.authc.token.enabled: true
    xpack.security.authc.api_key.enabled: true

After a start with systemctl start elasticsearch the Second, Third and Fourth node will automatically join the cluster due to the Cluster Discovery Seeding. To ensure the communication between Elasticsearch is encrypted I quick-check the communication with tshark:


    tshark -i eth0 -Y 'tcp.dstport==9200 and tls.app_data'

The output of tshark should be similar like this:


    1475 15.945403055 nnn.nnn.nnn.nnn → nnn.nnn.nnn.nnn TLSv1.2 598 Application Data
    1503 16.009221770 nnn.nnn.nnn.nnn → nnn.nnn.nnn.nnn TLSv1.2 1326 Application Data
    1604 17.102084056 nnn.nnn.nnn.nnn → nnn.nnn.nnn.nnn TLSv1.2 268 Application Data

Configuration of Kibana (First node)

The First node will also act as Kibana Dashboard for the data visualization. To install Kibana I’m following the standard installation process as described in Install Kibana with Debian package.

Because I’m operating my own Public key infrastructure for my home network/lab environment I can create inside the PKI a Certificate signing request and sign it directly with the Root certificate. The Certificate is then exported as PEM (Privacy-Enhanced Mail) format and transferred by SCP to the First node.

A possible /etc/kibana/kibana.yml configuration file might look like this:


    # ======================== Kibana Configuration =========================
    server.port: 5601
    server.host: 127.0.0.1
    server.name: "HOST_1.TLD"

    server.publicBaseUrl: "https://NGINX_HOST_1.TLD"

    # Enable TLS for Kibana with PKI-signed Certificate
    server.ssl.enabled: true
    server.ssl.certificate: /etc/kibana/HOST_1.TLD.pem
    server.ssl.key: /etc/kibana/HOST_1.TLD.pem

    server.securityResponseHeaders.strictTransportSecurity: "max-age=31536000"
    server.securityResponseHeaders.disableEmbedding: true
    csp.strict: true

    # Elasticsearch setup, use all four nodes
    elasticsearch.hosts: ["https://HOST_1.FQDN:9200", "https://HOST_2.FQDN:9200", "https://HOST_3.FQDN:9200", "https://HOST_4.FQDN:9200"]
    elasticsearch.username: "kibana_system"
    elasticsearch.ssl.certificateAuthorities: /etc/kibana/ELASTIC_SEARCH_CA.pem

    elasticsearch.pingTimeout: 1500
    elasticsearch.requestTimeout: 30000
    elasticsearch.shardTimeout: 30000

    kibana.index: ".kibana"
    kibana.defaultAppId: "home"

    # Set Xpack security settings
    xpack.security.enabled: true

    # Set to false if Kibana is monitored with Metricbeat module
    monitoring.kibana.collection.enabled: false

Configuration of Nginx as Reverse Proxy (First node)

Because Kibana is only reachable via localhost (127.0.0.1) I set up Nginx to act as a reverse proxy server between a web browser and Kibana. Nginx can be installed on Debian with the following command line:


    apt -y install nginx

Because no other web site will be hosted on this server, the configuration can be done directly in the /etc/nginx/sites-available/default file. The configuration includes a certificate signed by my own PKI and acts as a Reverse Proxy for Kibana (running on 127.0.0.1:5601). A configuration file for Nginx might look like this:


    server {
      listen IP_ADDR_1:443 ssl;
      server_name NGINX.TLD;
      index index.html;

      ssl_certificate /etc/ssl/private/NGINX.TLD.pem;
      ssl_certificate_key /etc/ssl/private/NGINX.TLD.pem;

      ssl_session_cache shared:SSL:10m;
      ssl_session_timeout 5m;
      ssl_protocols TLSv1.3;
      ssl_ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE
      ssl_prefer_server_ciphers on;

      location / {
        proxy_pass https://127.0.0.1:5601;
        proxy_read_timeout 90;
        proxy_connect_timeout 90;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Proxy "";
      }
    }

When the configuration is completed, Nginx can be restarted with:


    systemctl restart nginx

A quick test in a web browser should show the Elastic Welcome web page:

Elastic-Stack: Kibana via Nginx Reverse Proxy

Add GeoIP for Metricbeat with Kibana Dev Tools

To add GeoIP information like Country, City, etc. per IP addresss to Metricbeat, the Elastic-Stack must be able to connect to the public Elastic GeoIP endpoint (See GeoIP processor). With the Kibana Dev Tools a pipeline for GeoIP can be added (See Enrich events with geoIP information). The adding of the pipeline looks similar like this:

Elastic-Stack: Add GeoIP for Metricbeat with Kibana Dev Tools

Configuration of Metricbeat (First node)

To monitor the Elastic-Stack by itself, Metricbeat must have the modules beat-xpack, elasticsearch-xpack and system enabled. Because on the First node Kibana is installed, Kibana is monitored by Metricbeat as well. To activate the modules, the following command lines can be used:


    metricbeat modules enable beat-xpack
    metricbeat modules enable elasticsearch-xpack
    metricbeat modules enable system
    metricbeat modules enable kibana-xpack

A possible /etc/metricbeat/metricbeat.yml might look like the one shown below. Please note that this configuration is enriched with service tags to reflect the installed Metricbeat modules. The First node is also used to setup the Metricbeat dashboards in Kibana and requires therefore in the TLS configuration the Certificate of Kibana. Because the GeoIp pipeline was set up in Kibana Dev Tools, the pipeline can now be used by Metricbeat. The Elastic-Stack is monitoring itself (I’m not using a dedicated monitoring cluster) and therefore the monitoring.cluster_uuid must be set to itself.


    ###################### Metricbeat Configuration Example #######################
    metricbeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false

    setup.template.settings:
      index.number_of_shards: 1
      index.codec: best_compression

    name: "HOST_1.TLD"
    tags: ["service-metricbeat", "service-beat-xpack", "service-elasticsearch-xpack", "service-kibana-xpack", "service-nginx", "service-system"]
    fields:
      env: production

    setup.dashboards.enabled: false
    setup.kibana:
      host: "https://127.0.0.1:5601"
      username: "USERNAME"
      password: "PASSWORD"
      ssl:
        enabled: true
        certificate_authorities: ["/etc/metricbeat/ELASTIC_SEARCH_CA.pem", "/etc/kibana/KIBANA_HOST.TLD.pem"]
        verification_mode: "certificate"

    output.elasticsearch:
      hosts: ["HOST_1.TLD:9200", "HOST_2.TLD:9200", "HOST_3.TLD:9200", "HOST_4.TLD:9200"]
      protocol: "https"
      pipeline: geoip-info
      username: "USERNAME"
      password: "PASSWORD"
      ssl:
        enabled: true
        certificate_authorities: ["/etc/metricbeat/ELASTIC_SEARCH_CA.pem"]
        verification_mode: "certificate"

    processors:
      - add_host_metadata: ~
      - add_cloud_metadata: ~
      - add_docker_metadata: ~
      - add_kubernetes_metadata: ~

    logging.level: warning

    # Metricbeat internal HTTP endpoint (localhost only)
    http.enabled: true
    http.host: localhost
    http.port: 5066

Metricbeat module “system.yml” (First node)

A detailed description of this module is available at System module. The default configuration for the system.yml does not need any specific adjustments and therefore it might look like this:


    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
        - socket_summary

    process.include_top_n:
      by_cpu: 5
      by_memory: 5

    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)'

    - module: system
      period: 15m
      metricsets:
        - uptime

Metricbeat module “beat-xpack.yml” (First node)

A detailed description of this module is available at Beat module. I just make sure that xpack.enabled is set to true and that metricsets are removed as suggested in the documentation:


    - module: beat
      xpack.enabled: true
      period: 10s
      hosts: ["http://localhost:5066"]
      #username: "user"
      #password: "secret"

Metricbeat module “elasticsearch-xpack.yml” (First node)

A detailed description of this module is available at Elasticsearch module. In this module I set the hosts configuration to all four Elasticsearch nodes and add the certificate of the ELasticsearch CA in certificate_authorities. A possible configuration file might look like this:


    - module: elasticsearch
      xpack.enabled: true
      period: 10s
      hosts: ["https://HOST_1.TLD:9200", "https://HOST_2.TLD:9200", "https://HOST_3.TLD:9200", "https://HOST_4.TLD:9200"]
      username: "USERNAME"
      password: "PASSWORD"
      ssl:
        enabled: true
        certificate_authorities: ["/etc/metricbeat/ELASTIC_SEARCH_CA.pem"]
        verification_mode: "certificate"

Metricbeat module “kibana-xpack.yml” (First node)

A detailed description of this module is available at Kibana module. Because Kibana is only reachable through localhost (127.0.0.1) and with TLS encryption, the configuration file must be adjusted with settings for hosts and the host’s TLS certificate. A possible configuration might look like shown below:


    - module: kibana
      xpack.enabled: true
      period: 10s
      hosts: ["https://127.0.0.1:5601"]
      username: "USERNAME"
      password: "PASSWORD"
      ssl:
        enabled: true
        certificate_authorities: ["/etc/kibana/KIBANA_HOST.TLD.pem"]
        verification_mode: "certificate"

To setup the Metricbeat assets in Kibana I’m using the following command line:


    metricbeat setup -e

Configuration of Metricbeat (Second / Third / Fourth node)

The configuration of Metricbeat for the Second, Third and Fourth node are similar to the ones described in Configuration of Metricbeat (First node), Metricbeat module “system.yml” (First node), Metricbeat module “beat-xpack.yml” (First node) and Metricbeat module “elasticsearch-xpack.yml” (First node) except for the kibana-xpack module. Please note: Because those nodes are not used to load Metricbeat dashboards into Kibana, the configuration part for Kibana Dashboards is removed.

After a restart of Metricbeat with systemctl restart metricbeat the Cluster monitoring should show in the overview that Elasticsearch and Kibana is monitored with Metricbeat-monitoring as shown below:

Elastic-Stack: Cluster with Metricbeat self-monitoring

The screenshot shows that no Log data is found and for Log data Filebeat needs to be configured. This configuration will be done in one of my next blog posts.